The most surprising thing about GitLab CI debug traces is that they’re not just for debugging; they’re a fundamental tool for understanding the intent of your CI jobs.
Let’s see one in action. Imagine this simple .gitlab-ci.yml:
stages:
- build
build_job:
stage: build
script:
- echo "Building the project..."
- exit 1
When this job fails, GitLab’s default output is terse. It’ll show you the exit 1 and maybe the echo line. But what if the echo was part of a complex script? That’s where the debug trace comes in. To enable it, you don’t change your .gitlab-ci.yml. Instead, you go to your project’s Settings > CI/CD > Variables. Add a new variable:
- Key:
CI_DEBUG_TRACE - Value:
true - Type: Variable
- Protected: No (unless you’re using protected branches/tags and want it restricted)
- Masked: No
Now, re-run that pipeline. The job output will transform. Instead of just seeing the executed commands, you’ll see GitLab tracing the execution flow, including shell commands, variable expansions, and even the internal logic of the CI runner.
++ echo 'Building the project...'
Building the project...
++ exit 1
ERROR: Job failed: exit code 1
This ++ prefix is the key indicator of the debug trace. It signifies that the line is not just output from your script but a command being executed by the GitLab Runner’s shell.
The fundamental problem GitLab CI solves is automating software development workflows. It takes code changes, builds them, tests them, and deploys them without manual intervention. However, when these automated steps break, pinpointing the exact cause within potentially hundreds of lines of script and runner output can be like finding a needle in a haystack. The debug trace is the magnet. It reveals the sequence of commands the runner is executing, the arguments it’s passing, and the environment it’s operating in, all before your script’s own echo or print statements even get a chance to run.
Internally, when CI_DEBUG_TRACE is true, the GitLab Runner’s shell executor (or other executors that support it) wraps each command it’s about to execute with set -x. This shell option prints commands and their arguments as they are executed. The CI_DEBUG_TRACE variable is simply a flag that tells the GitLab Runner to enable this set -x behavior for all commands it runs within a job.
You control the level of detail by enabling or disabling this variable. There are no other levers for trace verbosity; it’s an on/off switch for the shell’s debugging feature.
Consider a more complex scenario:
build_job:
stage: build
script:
- export MY_VAR="initial_value"
- echo "Value before change: $MY_VAR"
- |
if [[ "$MY_VAR" == "initial_value" ]]; then
MY_VAR="changed_value"
fi
- echo "Value after change: $MY_VAR"
Without CI_DEBUG_TRACE, you’d see:
Value before change: initial_value
Value after change: changed_value
With CI_DEBUG_TRACE=true, you’d see:
++ export MY_VAR='initial_value'
++ echo 'Value before change: initial_value'
Value before change: initial_value
++ [[ initial_value == initial_value ]]
++ MY_VAR='changed_value'
++ echo 'Value after change: changed_value'
Value after change: changed_value
Notice how the ++ lines show the export, the echo, the [[ ... ]] conditional check, and the assignment MY_VAR='changed_value'. This level of detail is crucial for understanding why a condition might not be met or why a variable doesn’t hold the expected value at the point of execution. It shows the exact string being evaluated by [[ ... ]], revealing potential subtle differences in string content or whitespace that your script might otherwise obscure.
One aspect often missed is that CI_DEBUG_TRACE affects the runner’s interpretation of your script, not just the output of your script’s commands. For instance, if your script uses complex shell expansions or pipes, the debug trace will show the resolved command that the runner is actually trying to execute. This is invaluable when debugging issues related to quoting, variable interpolation, or command substitution that behave differently than you expect. It exposes the "final" command string as seen by the shell, which is often the source of subtle bugs.
The next step after mastering debug traces is understanding how to leverage GitLab’s built-in debugging features for specific executors, like the shell executor’s command option.