GitLab CI is failing to push images to the container registry because the Docker daemon on the CI runner is rejecting the authentication token.

Common Causes and Fixes

  1. Invalid or Expired Registry Token:

    • Diagnosis: Check the CI job logs for specific errors like denied: Token is invalid or unauthorized: authentication required. On the runner itself, you can try to manually pull an image using the CI-provided credentials. If the job logs show GET /v2/ or GET /v2/_catalog with a 401 status code, the token is likely the issue.
    • Fix: GitLab automatically generates a short-lived token for each job. If the job runs for an extended period, the token might expire before the push completes. Increase the CI_JOB_TOKEN_LIFETIME variable in your .gitlab-ci.yml file. For example:
      variables:
        CI_JOB_TOKEN_LIFETIME: 3600 # Set to 1 hour (in seconds)
      
      This extends the validity of the job token, allowing longer pushes.
    • Why it works: The CI_JOB_TOKEN_LIFETIME variable controls how long the token generated for the job remains valid. By increasing it, you ensure the token doesn’t expire during the image push operation.
  2. Incorrect Registry URL in CI Configuration:

    • Diagnosis: Examine the docker push command in your CI logs. Verify that the image tag includes the full registry URL, which for GitLab is typically registry.gitlab.com/your-group/your-project. A common mistake is omitting registry. or using an incorrect subdomain.
    • Fix: Ensure your docker build and docker push commands use the correct registry URL. A typical docker build command might look like this:
      docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA" .
      
      And the push command:
      docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
      
      The predefined CI/CD variable $CI_REGISTRY_IMAGE automatically resolves to the correct registry path for your project, like registry.gitlab.com/your-group/your-project.
    • Why it works: The Docker client needs the fully qualified image name to know which registry to authenticate against and push to. Using $CI_REGISTRY_IMAGE ensures this is correctly formatted.
  3. Runner Docker Daemon Not Authenticated to GitLab Registry:

    • Diagnosis: If you’re using a self-hosted runner or a runner with a custom Docker setup, the Docker daemon might not be configured to authenticate with your GitLab registry. Check runner logs for errors related to "authentication with registry" or "daemon not configured." You can also try manually logging in from the runner’s host: docker login registry.gitlab.com -u gitlab-ci-token -p $CI_JOB_TOKEN. If this fails with connection errors or authentication failures not related to token validity, the daemon might be the issue.
    • Fix: Ensure the Docker daemon on the runner has the necessary credentials. For GitLab.com, this is usually handled automatically when using the docker login command within the CI job. If you are on a self-hosted GitLab instance, you might need to configure the runner to use a specific user and token. The most common fix is to explicitly log in within the CI script before pushing:
      docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
      
      For GitLab.com, $CI_REGISTRY_USER is gitlab-ci-token and $CI_REGISTRY_PASSWORD is $CI_JOB_TOKEN. $CI_REGISTRY is registry.gitlab.com.
      docker login -u gitlab-ci-token -p "$CI_JOB_TOKEN" registry.gitlab.com
      
    • Why it works: The Docker daemon needs to authenticate with the registry using a valid username and password (or token) before it can push images. This explicit login command provides those credentials.
  4. Docker Daemon Outdated or Misconfigured:

    • Diagnosis: Older Docker versions might have compatibility issues with newer registry APIs or TLS configurations. Check the Docker version on your runner: docker version. Look for errors in the CI logs indicating TLS handshake failures, certificate issues, or protocol mismatches.
    • Fix: Update the Docker daemon on your runner to a recent stable version. For example, if you’re using a Linux VM, you’d typically use your package manager:
      sudo apt-get update
      sudo apt-get upgrade docker-ce docker-ce-cli containerd.io
      
      Ensure your Docker daemon is configured to use modern TLS protocols.
    • Why it works: Newer Docker versions have improved security and compatibility, resolving potential handshake or protocol errors that could prevent authentication.
  5. Network Connectivity or Firewall Issues:

    • Diagnosis: The runner might be unable to reach the GitLab container registry due to network restrictions. Check CI logs for timeouts during the docker push command or errors like connection refused or i/o timeout. You can test connectivity from the runner’s host: curl -v https://registry.gitlab.com.
    • Fix: Ensure that the runner’s network allows outbound connections to registry.gitlab.com on port 443 (HTTPS). If you are behind a proxy, configure your Docker daemon to use the proxy. For example, in /etc/systemd/system/docker.service.d/http-proxy.conf:
      [Service]
      Environment="HTTP_PROXY=http://your.proxy.server:port"
      Environment="HTTPS_PROXY=http://your.proxy.server:port"
      Environment="NO_PROXY=localhost,127.0.0.1,.gitlab.com"
      
      Then restart Docker: sudo systemctl daemon-reload && sudo systemctl restart docker.
    • Why it works: Network firewalls or misconfigured proxies can block the communication required for Docker to authenticate and push to the remote registry. Correcting these network settings allows the connection.
  6. Container Registry Storage Full (Self-Hosted GitLab):

    • Diagnosis: If you’re running a self-hosted GitLab instance, the underlying storage for the container registry might be full. Check your GitLab server’s disk space and the configured object storage (e.g., S3, GCS) usage. GitLab administrative logs might show errors related to disk full or object storage capacity.
    • Fix: Free up disk space on your GitLab server or increase the capacity of your object storage. For local storage, this might involve deleting old, unused images or increasing the disk size.
    • Why it works: The registry cannot store new image layers if its storage is exhausted, leading to push failures.

The next error you’ll likely encounter after fixing push failures is related to image pulling in subsequent stages, often manifesting as manifest unknown if the push partially succeeded or failed to complete cleanly.

Want structured learning?

Take the full Gitlab-ci course →