Flux reconciliation is not a single monolithic process; it’s a series of independent checks that run on separate schedules, and "pausing" one doesn’t affect the others.
Let’s watch this in action. Imagine we have a Flux reconciliation that’s taking too long, and we need to stop it without impacting our deployments. We’ll use the flux reconcile kustomization <name> --with-source command to trigger a reconciliation manually and then observe its behavior.
Here’s a Kustomization resource definition:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: my-app
namespace: flux-system
spec:
interval: 10m
path: ./clusters/my-app
sourceRef:
kind: GitRepository
name: my-git-repo
validation: client
When we run flux reconcile kustomization my-app --with-source --namespace flux-system, Flux will:
- Fetch the latest commit from the
my-git-repoGitRepository. - Apply the Kustomization located at
./clusters/my-appin that repository to the cluster. - Report the reconciliation status.
Now, if we need to pause this specific reconciliation before it starts, we can’t directly "pause" a running reconciliation with a flux command. Flux’s reconciliation mechanism is designed to be idempotent and event-driven. However, we can effectively "pause" future automatic reconciliations by temporarily modifying the interval and then resuming it later.
Let’s say we want to pause the my-app Kustomization’s automatic reconciliation for an hour. We’d run:
flux suspend kustomization my-app --namespace flux-system
This command sets a suspend: true annotation on the Kustomization resource. Flux controllers watch for this annotation and, if present, will skip any scheduled reconciliations for that resource.
When maintenance is complete, we resume it with:
flux resume kustomization my-app --namespace flux-system
This removes the suspend: true annotation, allowing the scheduler to pick it up again based on its interval.
The core problem Flux reconciliation solves is drift detection and correction. In a GitOps workflow, the cluster state should always reflect the desired state defined in Git. Reconciliation is the process that continuously verifies this and makes necessary changes. It’s not just about applying manifests; it’s about ensuring the source of truth (Git) is the source of reality (cluster). Each reconciliation cycle involves:
- Source Control Fetch: Fetching the latest desired state from the configured source (e.g., Git, Helm repository).
- Manifest Generation/Transformation: For Kustomizations, this means applying Kustomize overlays. For HelmReleases, it involves rendering the Helm chart.
- Drift Detection: Comparing the generated desired state with the actual state in the Kubernetes cluster.
- Application: If drift is detected, applying the necessary changes (create, update, delete resources) to the cluster to match the desired state.
- Status Reporting: Updating the status of the Flux resource (Kustomization, HelmRelease, etc.) to reflect the outcome of the reconciliation.
The flux suspend and flux resume commands directly manipulate the suspend annotation, which is a signal to the Flux controllers. The controllers, like the Kustomize Controller or Helm Controller, have an internal loop that checks for resources to reconcile. Part of this check includes looking for the suspend: true annotation. If it’s there, the controller simply skips that resource for that cycle without attempting any fetches or comparisons. This is why pausing one Kustomization doesn’t affect another; each resource is managed independently, and the suspend annotation is applied on a per-resource basis.
A common misconception is that flux suspend completely disables a resource. It doesn’t. It only prevents automatic reconciliations based on the interval. If you manually trigger a reconciliation using flux reconcile kustomization <name> --with-source, it will still run, even if the resource is suspended. This is a deliberate design choice, allowing for manual intervention when needed.
The next logical step after managing reconciliation schedules is understanding how to handle conflicts during reconciliation, especially when manual changes are made to the cluster that deviate from Git.