Jenkins is failing to write to its temporary directory because the underlying filesystem is full.
Here’s why that’s a problem: Jenkins uses temporary files for a lot of its operations, like unpacking archives, processing build logs, and managing workspaces. When the filesystem hosting these temporary files runs out of space, these operations fail abruptly, leading to build failures and a cascade of errors.
Common Causes and Fixes:
-
Build Artifacts and Logs Accumulating: Jenkins can retain old build artifacts and logs indefinitely if not configured otherwise. Over time, these can consume significant disk space.
- Diagnosis: Check the size of your Jenkins home directory, specifically
JENKINS_HOME/jobs/<job_name>/builds/andJENKINS_HOME/userContent/. - Fix: Implement a build retention strategy. In Jenkins, go to
Manage Jenkins->Configure System->Global properties->Build discarder. Select "Discard old builds" and choose a strategy like "Days to keep builds" (e.g.,7) or "Number of builds to keep" (e.g.,10). This will automatically clean up old build data. - Why it works: This policy tells Jenkins to periodically scan and remove build directories that exceed the specified retention criteria, freeing up disk space.
- Diagnosis: Check the size of your Jenkins home directory, specifically
-
Large Workspace Directories: If builds fail mid-execution, or if jobs are configured to not clean up their workspaces, these directories can grow enormous with downloaded dependencies, unpacked archives, and intermediate files.
- Diagnosis: Identify large workspace directories. These are typically found in
JENKINS_HOME/jobs/<job_name>/workspace/. Usedu -sh JENKINS_HOME/jobs/<job_name>/workspace/to check individual job workspace sizes. - Fix: Configure jobs to clean their workspaces before or after each build. In the job configuration, under "Build Environment," check "Delete workspace before build starts" or under "Post-build Actions," add a "Delete workspace when build is older than" option (e.g.,
7days). - Why it works: This ensures that stale or unnecessary files from previous builds are removed, preventing them from accumulating and consuming disk space.
- Diagnosis: Identify large workspace directories. These are typically found in
-
Temporary Directory (
/tmp) Overflow: The system’s temporary directory, often/tmp, is used by Jenkins and other processes. If it fills up, Jenkins will have nowhere to put its own temporary files.- Diagnosis: Check the free space on the partition hosting
/tmp. Rundf -h /tmp. If it’s near 100% usage, that’s your problem. - Fix:
- Option A (Clean up
/tmp): Manually remove old, unneeded files from/tmp. Be cautious, as other running processes might be using these files. A safer approach is to reboot the server, which often clears/tmp. - Option B (Configure Jenkins to use a different temp dir): Set the
java.io.tmpdirsystem property for Jenkins. This can be done by adding-Djava.io.tmpdir=/path/to/new/tempto theJAVA_ARGSin your Jenkins configuration file (e.g.,/etc/sysconfig/jenkinsorJENKINS_HOME/jenkins.xml). Ensure/path/to/new/tempexists and has ample space.
- Option A (Clean up
- Why it works: Either frees up space in the default temporary directory or redirects Jenkins’ temporary file creation to a partition with more available space.
- Diagnosis: Check the free space on the partition hosting
-
Docker Volume Bloat (if Jenkins uses Docker): If your Jenkins agents or builds run within Docker containers, the Docker storage itself can become a bottleneck. This includes images, volumes, and build caches.
- Diagnosis: Use
docker system dfto see Docker’s disk usage. Look for large volumes or dangling images. - Fix: Prune unused Docker resources. Run
docker system prune -a --volumes. This removes all stopped containers, all networks not used by at least one container, all dangling images, and all dangling build cache. The--volumesflag is crucial for removing unused volumes. - Why it works: This cleans up Docker’s internal storage, reclaiming space used by old images, containers, and unattached volumes that are no longer needed.
- Diagnosis: Use
-
System Logs Filling Up: Large or verbose system logs, especially those generated by Jenkins itself or its plugins, can consume disk space on the partition where Jenkins is installed or where logs are directed.
- Diagnosis: Check log file sizes. Common locations include
/var/log/jenkins/jenkins.logor withinJENKINS_HOME. Usedu -sh /var/log/jenkins/ordu -sh JENKINS_HOME/logs/. - Fix: Configure log rotation for Jenkins logs. Edit
/etc/logrotate.d/jenkins(or create it if it doesn’t exist) with a configuration like:
Then, run/var/log/jenkins/jenkins.log { daily rotate 7 compress missingok notifempty create 0640 jenkins adm }logrotate -f /etc/logrotate.d/jenkinsto apply immediately. - Why it works: Log rotation ensures that log files are periodically compressed and old ones are deleted, preventing them from growing indefinitely.
- Diagnosis: Check log file sizes. Common locations include
-
Large Plugin Data or Caches: Some Jenkins plugins store significant amounts of data or cache files within the
JENKINS_HOMEdirectory.- Diagnosis: Use
du -sh JENKINS_HOME/*to identify large directories within your Jenkins home. Look for directories associated with specific plugins (e.g.,JENKINS_HOME/plugins/<plugin_name>/,JENKINS_HOME/caches/). - Fix: Investigate the specific plugin. Some plugins have their own cleanup mechanisms or configuration options to manage their data size. For example, the "Artifact Manager" plugins might need configuration for artifact cleanup, or build scan plugins might need their cache limits adjusted.
- Why it works: By managing or cleaning the data generated by specific plugins, you reduce the overall disk footprint of the Jenkins installation.
- Diagnosis: Use
After resolving the disk space issue, you will likely encounter java.io.IOException: No space left on device errors related to Jenkins’ temporary directory.