Jenkins agents are the workhorses that actually run your builds, and scaling them is how you stop waiting for builds to finish.
Let’s see Jenkins agents in action. Imagine a simple Jenkinsfile:
pipeline {
agent any // This means Jenkins will pick any available agent
stages {
stage('Build') {
steps {
sh 'echo "Building on $(hostname)"'
}
}
stage('Test') {
steps {
sh 'echo "Testing on $(hostname)"'
}
}
}
}
When this pipeline runs, agent any tells Jenkins to find an available agent. If you only have one agent (your Jenkins controller, which is bad practice), builds queue up. If you have a few, they might still get busy. To scale, we need to tell Jenkins how to get more agents when needed.
The core problem Jenkins agents solve is separating the "brain" (the Jenkins controller) from the "brawn" (where builds actually execute). The controller manages jobs, schedules builds, and orchestrates, but it’s not designed to run compilation, testing, or deployment tasks. Distributing these tasks to agents provides several key benefits:
- Performance: Agents can be powerful machines, optimized for build tasks, without bogging down the controller.
- Isolation: Different projects might have conflicting dependencies or require specific operating systems. Agents provide isolated environments for each build.
- Scalability: As your build load increases, you can dynamically add more agents to handle the demand, rather than upgrading a single, monolithic controller.
- Cost-Effectiveness: For cloud-based Jenkins, you can spin up agents only when needed and shut them down afterward, saving on compute costs.
To achieve this, Jenkins uses a "controller-agent" architecture. The controller is the central Jenkins instance. Agents (formerly called "slaves") are separate machines that connect to the controller and wait for tasks.
Here’s how you set up a basic agent. First, on the machine you want to be an agent, install Java (the same version as your controller). Then, download the agent.jar file from your Jenkins controller. You can find this by navigating to YOUR_JENKINS_URL/jnlpJars/agent.jar.
On the agent machine, run the following command in your terminal:
java -jar agent.jar -jnlpUrl YOUR_JENKINS_URL/computer/YOUR_AGENT_NAME/slave-agent.jnlp -secret YOUR_AGENT_SECRET -workDir /path/to/jenkins/agent/workspace
YOUR_JENKINS_URL: This is the URL of your Jenkins controller (e.g.,http://localhost:8080).YOUR_AGENT_NAME: This is the name you’ll give your agent in Jenkins (e.g.,linux-build-node-01).YOUR_AGENT_SECRET: This is a security token Jenkins provides. You’ll get this when you configure the agent in the Jenkins UI./path/to/jenkins/agent/workspace: This is the directory on the agent machine where Jenkins will store build checkouts and temporary files.
In the Jenkins UI, you’d go to "Manage Jenkins" -> "Nodes" -> "New Node," give it a name, choose "Permanent Agent," and configure the launch method. For the manual agent.jar method, you’d select "Launch agent by connecting it to the master."
The agent.jar method is a good starting point, but it’s manual. For true scalability, you’ll want Jenkins to manage agent provisioning. This is where Cloud plugins come in. The most common ones are:
- EC2 Plugin: For provisioning agents on Amazon EC2.
- Docker Plugin: For spinning up Docker containers as agents.
- Kubernetes Plugin: For dynamic agent provisioning on Kubernetes clusters.
Let’s take the Kubernetes plugin as an example. You’d install it, then configure a "Cloud" in Jenkins under "Manage Jenkins" -> "Nodes" -> "New Cloud." You’ll point it to your Kubernetes API server, provide credentials, and define a "Pod Template."
A Pod Template specifies what kind of Kubernetes pod Jenkins should create for an agent. This includes the Docker image to use (e.g., jenkins/agent:latest), labels the agent will have (so your Jenkinsfile can target it), and environment variables.
Your Jenkinsfile would then look something like this to utilize these dynamic agents:
pipeline {
agent {
kubernetes {
label 'my-k8s-agent' // This label must match a label on your Pod Template
yaml """
apiVersion: v1
kind: Pod
spec:
containers:
- name: build-container
image: jenkins/agent:latest
command: ['cat'] # Keep the container running
tty: true
"""
}
}
stages {
stage('Build') {
steps {
sh 'echo "Building on $(hostname) inside Kubernetes pod"'
}
}
// ... other stages
}
}
When Jenkins needs an agent with the label my-k8s-agent, it tells the Kubernetes plugin. The plugin creates a pod based on your template, and once the agent.jar inside that pod connects back to Jenkins, your build can run on it. When the build finishes, Jenkins can be configured to automatically terminate the pod, freeing up resources.
The most surprising true thing about Jenkins agents is that the "agent" itself is just a Java executable (agent.jar) that establishes a persistent, bidirectional communication channel with the Jenkins controller. The controller sends build commands over this channel, and the agent executes them, sending back logs and results. This simple mechanism allows for immense flexibility, from simple SSH-connected machines to dynamic cloud instances.
When configuring your agent.jar launch command or your cloud provider’s agent templates, pay close attention to the workDir. If this directory is not accessible or writable by the user running the agent.jar process, or if it’s on a filesystem that gets unmounted between builds, your builds will fail with cryptic I/O errors or simply hang because Jenkins can’t check out code or write build artifacts.
The next step in scaling Jenkins is understanding how to define sophisticated agent requirements using labels and how to efficiently manage resource allocation in cloud environments.