The CI job failed because the GitLab Runner couldn’t properly check out your Git submodules. This is usually because the runner’s environment is missing the necessary credentials or configuration to access the submodule repositories.
Here are the most common reasons this happens and how to fix them:
1. Submodule Credentials Missing in Runner Environment
The runner needs to authenticate with the Git server to fetch submodules. If your submodules are in private repositories, the runner will fail if it doesn’t have the correct SSH key or access token.
-
Diagnosis:
- Manually try to clone the repository with submodules on a machine with the same OS and architecture as your GitLab Runner. If this fails with authentication errors, it’s a credential issue.
- Check your runner’s configuration. If it’s a shell executor, look for SSH keys in
~/.ssh/. If it’s a Docker executor, the image might not have the keys or credentials configured.
-
Fix (SSH Key):
- Generate an SSH key pair on your GitLab Runner machine (or a machine with the same setup).
ssh-keygen -t ed25519 -C "gitlab-runner-submodule" - Add the public key (
~/.ssh/id_ed25519.pub) as a Deploy Key to the parent GitLab project (the one containing the submodule). Grant it "Read-only" access if your CI only needs to fetch. - Ensure the private key (
~/.ssh/id_ed25519) is available to the runner process. For Docker executors, you’ll need to mount it as a volume or bake it into the image. For shell executors, ensure the user running thegitlab-runnerservice has read access to~/.ssh/id_ed25519. - In your
.gitlab-ci.yml, add abefore_scriptto configure SSH:before_script: - apt-get update -y && apt-get install -y openssh-client - eval "$(ssh-agent -s)" - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - - mkdir -p ~/.ssh - chmod 700 ~/.ssh - ssh-keyscan your.gitlab.host.com >> ~/.ssh/known_hosts - chmod 644 ~/.ssh/known_hosts$SSH_PRIVATE_KEYshould be a GitLab CI/CD variable (Settings > CI/CD > Variables) containing the content of your private SSH key.- Replace
your.gitlab.host.comwith your GitLab instance’s hostname.
- Generate an SSH key pair on your GitLab Runner machine (or a machine with the same setup).
-
Why it works: This makes the runner’s environment aware of the SSH key and adds your GitLab server to its list of trusted hosts, allowing it to connect securely to fetch the submodule.
-
Fix (HTTPS Token):
- Create a Personal Access Token (PAT) in GitLab with
read_repositoryscope for the user whose repositories are being used as submodules. - In your
.gitlab-ci.yml, use the token in the submodule URL. You can store the token in a CI/CD variable (e.g.,GITLAB_TOKEN).before_script: - git config --global url."https://oauth2:${GITLAB_TOKEN}@your.gitlab.host.com/".insteadOf "https://your.gitlab.host.com/"- Replace
your.gitlab.host.comwith your GitLab instance’s hostname. GITLAB_TOKENshould be a GitLab CI/CD variable.
- Replace
- Create a Personal Access Token (PAT) in GitLab with
-
Why it works: This rewrites the submodule URLs on the fly, injecting the authentication token so Git can access private repositories over HTTPS.
2. Incorrect Submodule URL Configuration
The .gitmodules file might contain an incorrect or outdated URL for the submodule.
-
Diagnosis:
- Examine the
.gitmodulesfile in your project’s root. Verify theurlparameter for each submodule. - Try cloning the parent repository and then manually running
git submodule update --init --recursiveon your local machine. If it fails, the.gitmodulesfile is likely the culprit.
- Examine the
-
Fix:
- Correct the URL in your
.gitmodulesfile to point to the exact repository location. - Commit and push the
.gitmodulesfile. - In your
.gitlab-ci.yml, ensure thegit fetchorgit clonecommands are configured to use the correct strategy. The defaultgit checkoutin GitLab CI usually handles submodules if they are correctly configured. If you’re manually cloning, ensure it’s done with submodule initialization:script: - git clone --recurse-submodules $CI_REPOSITORY_URL . # or if already cloned - git submodule update --init --recursive
- Correct the URL in your
-
Why it works: Git uses the
.gitmodulesfile to know where to find and how to clone submodules. An incorrect URL means Git can’t locate the submodule repository.
3. Runner Not Initializing Submodules
By default, GitLab CI attempts to check out submodules. However, if you’ve customized the GIT_STRATEGY or are using a specific GIT_SUBMODULE_STRATEGY, it might be misconfigured.
-
Diagnosis:
- Check your
.gitlab-ci.ymlforGIT_STRATEGYorGIT_SUBMODULE_STRATEGYvariables. - Look at the CI job logs for Git commands that might be skipping submodule initialization.
- Check your
-
Fix:
- Ensure
GIT_SUBMODULE_STRATEGYis set correctly. The common options are:normal: Initialize and update submodules.recursive: Initialize and update submodules recursively.none: Do not initialize submodules.
- Set it in your
.gitlab-ci.ymlglobally or per job:variables: GIT_SUBMODULE_STRATEGY: recursive your_job_name: script: - echo "Submodules should be checked out now."
- Ensure
-
Why it works: This explicitly tells the GitLab Runner how to handle submodules during the checkout process.
recursiveis often the safest bet.
4. Git Version Incompatibility
Older versions of Git on the runner might have issues with newer submodule features or authentication methods.
-
Diagnosis:
- Check the Git version on your runner:
git --version - Compare this to the Git version used on a working local machine.
- Check the Git version on your runner:
-
Fix:
- Update Git on your runner.
- If using Docker executors, update your Docker image to one with a recent Git version. For example, use a newer Ubuntu base image or a dedicated Git image.
- If using shell executors, update Git via your system’s package manager (e.g.,
sudo apt-get update && sudo apt-get install giton Debian/Ubuntu).
-
Why it works: Newer Git versions include bug fixes and support for modern cryptographic protocols and authentication mechanisms that older versions might lack.
5. Submodule Repository is Unavailable or Corrupt
The submodule repository itself might be temporarily down, deleted, or in a corrupted state.
-
Diagnosis:
- Attempt to clone the submodule repository directly from your local machine using its URL.
- Check the status of the GitLab instance hosting the submodule.
-
Fix:
- If the submodule repository is unavailable, wait for it to be restored.
- If the repository was deleted, restore it or update your project to point to a different, available submodule.
- If the submodule repository is corrupt, attempt to repair it on the hosting server.
-
Why it works: The CI job cannot fetch a submodule that doesn’t exist or is inaccessible.
6. Network Restrictions or Firewalls
The runner might be in a network environment that blocks outbound connections to the Git server hosting your submodules.
-
Diagnosis:
- From the runner’s environment, try to
pingorcurlthe hostname of the submodule’s Git server. - Check firewall rules on the runner’s host machine or any network infrastructure between the runner and the Git server.
- From the runner’s environment, try to
-
Fix:
- Configure your network firewall to allow outbound connections on port 22 (for SSH) or port 443 (for HTTPS) to the Git server hosting your submodules.
- If the runner is in a restricted corporate network, you might need to consult your network administrators.
-
Why it works: Network restrictions prevent the runner from establishing the necessary connection to download the submodule code.
After applying these fixes, your next potential issue might be a failure in a subsequent job that relies on the checked-out submodules, such as a build step that compiles code within those submodules.