Jenkins is silently consuming disk space, and your builds are starting to fail with java.io.IOException: No space left on device. This isn’t just about old build logs; it’s about the core Jenkins workspace and the Jenkins home directory growing unchecked, starving the OS of critical space needed for temporary files and basic operations.
Let’s break down the common culprits and how to tackle them.
1. Stale Build Artifacts and Workspace Bloat
Jenkins keeps a lot of data for each build: logs, workspace files, test reports, and artifacts. Over time, this accumulates.
Diagnosis: Navigate to your Jenkins home directory (often /var/lib/jenkins). Inside, you’ll find a jobs directory. Each subdirectory represents a Jenkins job. Within a job’s directory, look for a builds folder. This is where individual build data resides. You can also check workspace directories directly.
# Example: Check space used by a specific job's builds
du -sh /var/lib/jenkins/jobs/MyJob/builds/*
Fix: Jenkins has built-in mechanisms to prune old builds. Configure this in your job’s settings.
- Go to your Jenkins job configuration.
- Under "Advanced build options," find "Discard old builds."
- Set "Days to keep builds" to a reasonable number (e.g., 7 or 14).
- Set "Number of builds to keep" to a similar number (e.g., 10 or 20).
This tells Jenkins to automatically delete build data older than the specified days or number of builds, freeing up space.
2. Unused Jenkins Plugins and their Data
Plugins can be fantastic, but they also consume disk space, sometimes by storing their own data within the Jenkins home directory. If you’ve uninstalled plugins without cleaning up their associated data, it can linger.
Diagnosis: Examine the plugins directory within your Jenkins home. Also, look for directories named after plugins or their developers in the root of your Jenkins home.
# Check space used by plugins directory
du -sh /var/lib/jenkins/plugins
Fix: Regularly review your installed plugins. Uninstall any that are no longer needed. Jenkins usually cleans up associated plugin files upon uninstallation, but sometimes residual data can remain. A restart of Jenkins after uninstalling plugins is often a good idea.
3. Jenkins Controller’s tmp Directory
Jenkins, like any Java application, uses temporary files. These are often stored in the system’s default temporary directory (e.g., /tmp on Linux) or a Jenkins-specific temporary directory. If these aren’t cleaned up, they can grow quite large.
Diagnosis: Check the size of your system’s /tmp directory or any Jenkins-specific temp directories.
# Check default system temp directory
du -sh /tmp
Fix: Configure Jenkins to clean its own temporary directory. You can set a system property for Jenkins to use a specific temporary directory and then set up a cron job to clean it.
-
Jenkins Configuration (optional, but good practice):
- Edit your Jenkins startup script (e.g.,
/etc/default/jenkinsorjenkins.sh). - Add or modify
JAVA_ARGSto include-Djava.io.tmpdir=/var/lib/jenkins/tmp:JAVA_ARGS="-Djava.io.tmpdir=/var/lib/jenkins/tmp -Djava.awt.headless=true" - Create the directory:
sudo mkdir -p /var/lib/jenkins/tmp && sudo chown jenkins:jenkins /var/lib/jenkins/tmp - Restart Jenkins.
- Edit your Jenkins startup script (e.g.,
-
Cron Job:
- Create a cron job to clean this directory periodically. For example, to clean it daily:
# Add to crontab -e 0 3 * * * find /var/lib/jenkins/tmp -type f -mtime +7 -delete
This command deletes files in
/var/lib/jenkins/tmpthat haven’t been modified in the last 7 days. - Create a cron job to clean this directory periodically. For example, to clean it daily:
This ensures Jenkins’ temporary files are managed and don’t exhaust disk space.
4. Build Workspace Cleanup
Even with "Discard old builds" enabled, the workspace itself can accumulate files if jobs don’t clean up after themselves, or if build steps leave large temporary files.
Diagnosis: Examine the workspace directory for your jobs.
# Check space used by a specific job's workspace
du -sh /var/lib/jenkins/workspace/MyJob
Fix: Configure your jobs to clean the workspace before building.
- In your Jenkins job configuration, under "Build Environment," check "Delete workspace before build starts."
This ensures that each build starts with a clean slate, preventing accumulation of old build artifacts within the workspace.
5. Jenkins Controller Disk Full (System Level)
Sometimes, the issue isn’t just Jenkins data, but the entire disk partition where Jenkins resides is full. This can happen due to other system processes, log files, or large application data outside of Jenkins.
Diagnosis: Use df -h to check overall disk usage on your Jenkins server.
df -h
Look for partitions that are at or near 100% usage. Pay close attention to the partition containing /var/lib/jenkins.
Fix: This is a broader system administration task. You’ll need to identify what’s consuming the space on that partition. This might involve:
- Cleaning up old system logs (
/var/log). - Removing old Docker images or containers if applicable.
- Identifying and removing large, unneeded files.
- Expanding the disk size or moving Jenkins data to a larger partition.
6. Large Artifacts and External Storage
If your builds produce very large artifacts, and these are being stored on the Jenkins controller, they can quickly consume space.
Diagnosis: Check the size of artifact directories if you’re using Jenkins’ built-in artifact archiving.
# Example if artifacts are stored within job builds
du -sh /var/lib/jenkins/jobs/MyJob/builds/*/archive
Fix: Consider storing large artifacts externally (e.g., in an S3 bucket, Nexus, Artifactory) rather than relying on Jenkins’ built-in archiving. Update your build jobs to upload artifacts to external storage and configure Jenkins to only retain build status information.
The next error you’ll likely encounter if you fix disk space issues but have other underlying problems is a FileNotFoundException during build execution, as Jenkins might struggle to access or write to its own directories.