GitLab CI pipelines are so flexible they can actually be used to build the YAML parser that validates them.
Let’s see a basic pipeline in action. Imagine you have a .gitlab-ci.yml file in your repository like this:
stages:
- build
- test
- deploy
build_job:
stage: build
script:
- echo "Building the project..."
- make build
artifacts:
paths:
- build/
test_job:
stage: test
script:
- echo "Running tests..."
- make test
dependencies:
- build_job
deploy_job:
stage: deploy
script:
- echo "Deploying the application..."
- deploy --production
when: manual
needs:
- test_job
When a commit is pushed, GitLab Runner picks up this configuration. It sees three stages: build, test, and deploy. Jobs are grouped by stage and executed in that order. The build_job runs first. Its script section contains two commands that will be executed sequentially in a temporary environment provided by the Runner. If these commands succeed, any files in the build/ directory are uploaded as artifacts, making them available for subsequent jobs.
Next, the test_job runs. It also has a script section. Crucially, it has dependencies: - build_job. This tells the Runner to download the artifacts produced by build_job before executing its own script. This is how you pass outputs from one job to another.
Finally, the deploy_job is defined. Notice when: manual. This means the job won’t run automatically after test_job completes. Instead, a user will have to manually click a "play" button in the GitLab UI to trigger it. The needs: - test_job directive is a more granular way to define dependencies than stages, allowing for more complex DAGs (Directed Acyclic Graphs) where jobs can run in parallel if they don’t depend on each other, or skip stages entirely.
The core of GitLab CI’s power lies in its YAML syntax. You define jobs which are the fundamental units of work. Each job has a script (the commands to run) and can belong to a stage. You can define variables for configuration, use cache to speed up subsequent runs by sharing files between jobs, and specify rules or only/except to control when a job runs (e.g., only on specific branches or tags). The image keyword specifies the Docker image the job will run inside, providing a consistent and isolated environment. For example, image: ruby:2.7 would run the job within a Docker container using the Ruby 2.7 image.
The gitlab-ci.yml file is parsed by GitLab’s CI/CD system, which then communicates with GitLab Runners to execute the defined jobs. Runners are separate agents that perform the actual work. When a job is triggered, GitLab sends the job details to an available Runner. The Runner pulls the specified Docker image (or uses a shell executor), mounts any necessary volumes, and executes the script commands.
The way needs and dependencies interact is a common point of confusion. dependencies is the older mechanism and it downloads all artifacts from the specified jobs. needs is more modern and allows for finer-grained control, including the ability to specify artifacts: { paths: [...] } within the needs directive itself, meaning you only download the specific files you require, not everything. This can significantly reduce job setup time.
When you define a service within a job, like services: - postgres:12, GitLab will start a secondary Docker container (in this case, a PostgreSQL 12 instance) alongside your main job container. This is incredibly useful for integration testing, where your application might need to connect to a database or another service. The service container is accessible by hostname (e.g., postgres) from within your job container.
The timeout keyword for a job is set in minutes, and if the job exceeds this duration, it will be automatically cancelled by the Runner. For example, timeout: 30 means the job will run for a maximum of 30 minutes. If a job consistently times out, it usually indicates an inefficient script, a need for more resources, or a potential deadlock. You can also set a global timeout for the entire pipeline in the .gitlab-ci.yml file at the root level.
The trigger keyword allows you to create multi-project pipelines, where one pipeline can initiate another pipeline in a different GitLab project. This is essential for breaking down large, complex systems into manageable, independently deployable components. You specify the project and branch to trigger.
The relationship between rules and only/except is that rules is the newer, more flexible syntax. While only/except are still supported for backward compatibility, rules allows for more complex conditions using if, changes, exists, and when keywords. For instance, rules: - if: '$CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"' would ensure a job only runs for pushes to the main branch.
The most surprising thing about GitLab CI is that you can define workflow rules at the top level of your .gitlab-ci.yml to control pipeline creation itself, independent of individual job rules. This allows you to, for example, prevent pipelines from running for certain branches or based on specific pipeline triggers, effectively short-circuiting the entire CI process before any jobs are even considered.
The next concept you’ll likely encounter is how to manage secrets and sensitive information securely within your pipelines, often involving GitLab’s built-in CI/CD variables with masking and protected settings.