GitLab CI is failing to push images to the container registry because the Docker daemon on the CI runner is rejecting the authentication token.
Common Causes and Fixes
-
Invalid or Expired Registry Token:
- Diagnosis: Check the CI job logs for specific errors like
denied: Token is invalidorunauthorized: authentication required. On the runner itself, you can try to manually pull an image using the CI-provided credentials. If the job logs showGET /v2/orGET /v2/_catalogwith a 401 status code, the token is likely the issue. - Fix: GitLab automatically generates a short-lived token for each job. If the job runs for an extended period, the token might expire before the push completes. Increase the
CI_JOB_TOKEN_LIFETIMEvariable in your.gitlab-ci.ymlfile. For example:
This extends the validity of the job token, allowing longer pushes.variables: CI_JOB_TOKEN_LIFETIME: 3600 # Set to 1 hour (in seconds) - Why it works: The
CI_JOB_TOKEN_LIFETIMEvariable controls how long the token generated for the job remains valid. By increasing it, you ensure the token doesn’t expire during the image push operation.
- Diagnosis: Check the CI job logs for specific errors like
-
Incorrect Registry URL in CI Configuration:
- Diagnosis: Examine the
docker pushcommand in your CI logs. Verify that the image tag includes the full registry URL, which for GitLab is typicallyregistry.gitlab.com/your-group/your-project. A common mistake is omittingregistry.or using an incorrect subdomain. - Fix: Ensure your
docker buildanddocker pushcommands use the correct registry URL. A typicaldocker buildcommand might look like this:
And the push command:docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
The predefined CI/CD variabledocker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"$CI_REGISTRY_IMAGEautomatically resolves to the correct registry path for your project, likeregistry.gitlab.com/your-group/your-project. - Why it works: The Docker client needs the fully qualified image name to know which registry to authenticate against and push to. Using
$CI_REGISTRY_IMAGEensures this is correctly formatted.
- Diagnosis: Examine the
-
Runner Docker Daemon Not Authenticated to GitLab Registry:
- Diagnosis: If you’re using a self-hosted runner or a runner with a custom Docker setup, the Docker daemon might not be configured to authenticate with your GitLab registry. Check runner logs for errors related to "authentication with registry" or "daemon not configured." You can also try manually logging in from the runner’s host:
docker login registry.gitlab.com -u gitlab-ci-token -p $CI_JOB_TOKEN. If this fails with connection errors or authentication failures not related to token validity, the daemon might be the issue. - Fix: Ensure the Docker daemon on the runner has the necessary credentials. For GitLab.com, this is usually handled automatically when using the
docker logincommand within the CI job. If you are on a self-hosted GitLab instance, you might need to configure the runner to use a specific user and token. The most common fix is to explicitly log in within the CI script before pushing:
For GitLab.com,docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY$CI_REGISTRY_USERisgitlab-ci-tokenand$CI_REGISTRY_PASSWORDis$CI_JOB_TOKEN.$CI_REGISTRYisregistry.gitlab.com.docker login -u gitlab-ci-token -p "$CI_JOB_TOKEN" registry.gitlab.com - Why it works: The Docker daemon needs to authenticate with the registry using a valid username and password (or token) before it can push images. This explicit login command provides those credentials.
- Diagnosis: If you’re using a self-hosted runner or a runner with a custom Docker setup, the Docker daemon might not be configured to authenticate with your GitLab registry. Check runner logs for errors related to "authentication with registry" or "daemon not configured." You can also try manually logging in from the runner’s host:
-
Docker Daemon Outdated or Misconfigured:
- Diagnosis: Older Docker versions might have compatibility issues with newer registry APIs or TLS configurations. Check the Docker version on your runner:
docker version. Look for errors in the CI logs indicating TLS handshake failures, certificate issues, or protocol mismatches. - Fix: Update the Docker daemon on your runner to a recent stable version. For example, if you’re using a Linux VM, you’d typically use your package manager:
Ensure your Docker daemon is configured to use modern TLS protocols.sudo apt-get update sudo apt-get upgrade docker-ce docker-ce-cli containerd.io - Why it works: Newer Docker versions have improved security and compatibility, resolving potential handshake or protocol errors that could prevent authentication.
- Diagnosis: Older Docker versions might have compatibility issues with newer registry APIs or TLS configurations. Check the Docker version on your runner:
-
Network Connectivity or Firewall Issues:
- Diagnosis: The runner might be unable to reach the GitLab container registry due to network restrictions. Check CI logs for timeouts during the
docker pushcommand or errors likeconnection refusedori/o timeout. You can test connectivity from the runner’s host:curl -v https://registry.gitlab.com. - Fix: Ensure that the runner’s network allows outbound connections to
registry.gitlab.comon port 443 (HTTPS). If you are behind a proxy, configure your Docker daemon to use the proxy. For example, in/etc/systemd/system/docker.service.d/http-proxy.conf:
Then restart Docker:[Service] Environment="HTTP_PROXY=http://your.proxy.server:port" Environment="HTTPS_PROXY=http://your.proxy.server:port" Environment="NO_PROXY=localhost,127.0.0.1,.gitlab.com"sudo systemctl daemon-reload && sudo systemctl restart docker. - Why it works: Network firewalls or misconfigured proxies can block the communication required for Docker to authenticate and push to the remote registry. Correcting these network settings allows the connection.
- Diagnosis: The runner might be unable to reach the GitLab container registry due to network restrictions. Check CI logs for timeouts during the
-
Container Registry Storage Full (Self-Hosted GitLab):
- Diagnosis: If you’re running a self-hosted GitLab instance, the underlying storage for the container registry might be full. Check your GitLab server’s disk space and the configured object storage (e.g., S3, GCS) usage. GitLab administrative logs might show errors related to disk full or object storage capacity.
- Fix: Free up disk space on your GitLab server or increase the capacity of your object storage. For local storage, this might involve deleting old, unused images or increasing the disk size.
- Why it works: The registry cannot store new image layers if its storage is exhausted, leading to push failures.
The next error you’ll likely encounter after fixing push failures is related to image pulling in subsequent stages, often manifesting as manifest unknown if the push partially succeeded or failed to complete cleanly.