The exec format error on multi-architecture Kubernetes nodes means the container runtime on the node tried to execute a binary, but the binary’s instruction set architecture doesn’t match the CPU architecture of the node.

Here are the common causes and how to fix them:

1. Incorrect Image Architecture for the Node

This is the most frequent culprit. You’ve pulled an image built for, say, amd64 (Intel/AMD) and are trying to run it on an arm64 (AWS Graviton, Raspberry Pi) node, or vice-versa.

  • Diagnosis:
    • Find the node experiencing the error: kubectl get pods -o wide | grep <your-pod-name>
    • Get the node’s architecture: kubectl get node <node-name> -o jsonpath='{.status.nodeInfo.architecture}'
    • Inspect the image your pod is trying to use. If you’re using docker.io/library/nginx:latest, for example, you need to know what architecture that latest tag resolves to on your registry for the specific node’s architecture. The best way is to check the manifest list: docker manifest inspect <your-image-name>:<tag> Look for the architecture field in the manifests array.
  • Fix:
    • Option A (Recommended): Use an image with explicit multi-arch support. Many official images (like nginx, ubuntu, redis) are built as multi-arch. When you pull nginx:latest on an amd64 node, docker (or containerd) automatically pulls the amd64 variant. If you’re seeing this error, it’s likely you’re not using a multi-arch image, or your registry/runtime is misconfigured.
    • Option B: Specify the architecture in your image tag. If you know you need amd64, explicitly tag it: your-repo/your-image:1.0-amd64. If you need arm64: your-repo/your-image:1.0-arm64. You’ll need to build and push these separately.
    • Option C: Use image pull policies carefully. If you’re using imagePullPolicy: Always and your registry is serving an incorrect architecture because of a latest tag resolution issue, this can happen. Consider using specific, immutable tags.
  • Why it works: The container runtime on the node will only download and attempt to execute binaries compiled for its own CPU architecture. By ensuring the image you’re pulling contains a manifest entry for the node’s architecture, the runtime can select the correct variant.

2. amd64 Container on arm64 Node (or vice-versa) via containerd configuration

Sometimes, even with multi-arch images, the containerd configuration might be forcing a specific architecture or not correctly resolving multi-arch manifests.

  • Diagnosis:
    • Check the containerd configuration on the node, typically at /etc/containerd/config.toml.
    • Look for settings related to image handling or architecture preferences.
  • Fix:
    • Ensure containerd is configured to correctly handle multi-arch manifests. For most modern containerd versions, this is the default. If you’ve customized it, review your [plugins."io.containerd.grpc.v1.cri".registry.mirrors] or other relevant sections.
    • Restart containerd: sudo systemctl restart containerd
  • Why it works: containerd is responsible for pulling and managing images. If its configuration interferes with the standard multi-arch manifest resolution, restarting it with a correct default configuration allows it to pick the appropriate architecture.

3. amd64 Container on arm64 Node (or vice-versa) via docker daemon configuration (if using dockershim)

If your Kubernetes cluster is older and still using the deprecated dockershim, the docker daemon itself might be misconfigured.

  • Diagnosis:
    • Check the docker daemon configuration on the node, typically in /etc/docker/daemon.json.
    • Look for any settings that might be restricting image architectures.
  • Fix:
    • Ensure no explicit architecture restrictions are set in daemon.json. The default behavior is to pull the architecture matching the host.
    • Restart the docker daemon: sudo systemctl restart docker
  • Why it works: Similar to containerd, the docker daemon is the image manager. A misconfiguration here can lead it to fetch or attempt to run the wrong architecture.

4. Corrupted Image Cache on the Node

A corrupted image layer or manifest can lead to the container runtime failing to extract or execute parts of the image.

  • Diagnosis:
    • Try to explicitly pull the image on the affected node:
      • If using containerd: sudo ctr images pull <your-image-name>:<tag>
      • If using docker: sudo docker pull <your-image-name>:<tag>
    • If the pull fails or exhibits strange behavior, the cache might be an issue.
  • Fix:
    • Remove the problematic image from the node’s cache:
      • If using containerd: sudo ctr images rm <your-image-name>:<tag>
      • If using docker: sudo docker rmi <your-image-name>:<tag>
    • Then, let Kubernetes re-pull the image.
  • Why it works: By removing the corrupted local copy, the container runtime is forced to download a fresh, uncorrupted version of the image from the registry.

5. Binary Not Compiled for the Target Architecture (Application Level)

This is less common if you’re using official multi-arch images but can happen with custom-built binaries within an image. You might have an image that itself is multi-arch (e.g., it has both amd64 and arm64 binaries), but the specific binary your ENTRYPOINT or CMD is trying to run is only compiled for the wrong architecture, even though the image could technically support the node.

  • Diagnosis:
    • Exec into a container of the same image on a correct architecture node and inspect the binary: kubectl exec -it <pod-name> -- /bin/sh file /path/to/your/binary This will tell you the architecture of the binary.
    • Compare this to the node’s architecture you found in cause #1.
  • Fix:
    • Rebuild your application binary and include it in your Dockerfile for the target architecture.
    • If you’re using a base image, ensure the base image’s binaries are also multi-arch or that you’re using an architecture-specific variant of the base image.
  • Why it works: The exec format error happens when the kernel tries to load a program that it cannot understand. If the binary itself is for the wrong CPU type, it will fail regardless of the image’s multi-arch capabilities.

6. Node Architecture Mismatch in Pod nodeSelector or affinity

Less likely to cause exec format error directly, but can lead you to troubleshoot the wrong node. If you’ve incorrectly configured your pod to only schedule on a specific architecture node that doesn’t exist or is misidentified.

  • Diagnosis:
    • Check your Pod/Deployment spec for nodeSelector or nodeAffinity rules that might be targeting a specific architecture label (e.g., kubernetes.io/arch: arm64).
    • Verify the labels on your nodes: kubectl get nodes --show-labels
  • Fix:
    • Correct the nodeSelector or nodeAffinity rules to accurately reflect the available node architectures.
  • Why it works: This ensures the pod lands on a node where it’s expected to run, preventing the runtime from even attempting to execute an incompatible binary.

The next error you’ll likely encounter if you’ve fixed the architecture mismatch but there’s still an issue with image pulling is a ImagePullBackOff or ErrImagePull, indicating that the registry is inaccessible or the image name/tag is incorrect.

Want structured learning?

Take the full Kubernetes course →