GitOps & continuous delivery

Why kubectl apply in a pipeline is not GitOps

Your prerequisite, ci-cd-pipelines, ends where the image exists: built, tested, tagged, sitting in a registry. GitOps is one answer to what happens next, and it is a specific one. Putting manifests in git and having CI run kubectl apply at the end is push delivery, not GitOps.

OpenGitOps (the CNCF project that pins the definition) lists four principles. The state is declarative, it is versioned and immutable in git, an agent pulls it automatically, and that agent continuously reconciles live state toward it. A push pipeline satisfies the first two and skips the last two: it fires once per commit, applies, and stops watching. GitOps runs a controller inside the cluster that keeps comparing what git says to what is actually running.

The payoff of the loop is that git becomes the source of truth about what is running, updated continuously as the cluster changes. The cost is a new component to run and reason about.

Drift, and the two switches that handle it

Drift is when live state stops matching git: someone runs kubectl edit, an operator mutates a field, a node dies and something reschedules oddly. The controller notices on its next pass and flags the app OutOfSync.

Noticing is not fixing. Detecting drift and correcting it are separate settings, and confusing them burns people:

Wrong "We turned on automated sync, so anything I change by hand in prod snaps back to git." Automated sync applies git changes for you and marks live drift OutOfSync, but reverting a live-only change is a second switch, self-heal, off by default in Argo CD. With self-heal off you see the drift and it sits there until a git commit triggers the next sync. Right enable self-heal if you want the cluster pulled back to git automatically (Argo CD re-syncs on a 5-second self-heal timeout). Keep it off if operators legitimately need to hand-tune and you would rather be told than overruled.

Pruning is a third, independent switch: whether the controller deletes resources you removed from git. It ships off as a safety default too, so a resource dropped from git lingers until you enable prune.

Argo CD and Flux, the two-minute version

Both watch git and reconcile a cluster. They differ in shape.

Argo CD centers on the Application, a custom resource that says "this git path renders to these Kubernetes resources, sync them here." You mostly drive it through a UI and CLI, and it is comfortable managing many clusters from one control plane. To bootstrap a whole cluster you use app-of-apps: one parent Application whose job is to create other Applications.

Flux is a set of controllers (the GitOps Toolkit) rather than one app. A GitRepository (or OCIRepository) is the source; a Kustomization or a HelmRelease is the thing reconciled from it. There is no central UI in the box; you drive it by committing custom resources. Flux leans Kubernetes-native and composable.

Choose on operating model rather than feature lists: a platform team serving many teams from one pane often likes Argo CD; a team that wants delivery to be just more Kubernetes resources often likes Flux. Both are healthy CNCF-graduated projects as of mid-2026 (Argo CD around v3.4, Flux around v2.8).

Rollback is a git revert, and secrets do not go in git raw

When a release is bad, the reflex from push pipelines is "redeploy the old build." In GitOps the rollback is git revert of the bad commit. That makes the known-good state the new HEAD, and the controller reconciles the cluster back to it on the next pass. You get a normal commit with an author and a timestamp, so the rollback is in the history like any other change.

Both Argo CD and Flux offer a rollback command, but it moves the cluster without moving git, which leaves them out of step. Reach for revert first; the command is for emergencies where you accept the divergence.

Secrets are the one thing you do not commit as-is:

Wrong "Kubernetes Secrets are base64, so committing them to the repo is fine." base64 only encodes; it does not encrypt. Anyone with read access to the repo runs one command and has your credentials. Right either encrypt the secret before it enters git (SOPS, Sealed Secrets) or keep only a pointer in git and let a controller pull the real value from a vault at runtime (External Secrets Operator). The mechanics of KMS, leases, and rotation belong to the secrets-management topic; here the rule is just that plaintext or base64 credentials never land in the config repo.

Pull's real payoff, and the side-channels that quietly defeat it

The reason to run a reconciler instead of a push step is not fashion. A push pipeline knows the cluster only at apply time; between deploys it is blind. A reconciler observes continuously, so git stops being "what we deployed once" and becomes a live claim about what is running that you can diff, alert on, and enforce.

That claim is only as strong as your discipline about back doors. Every out-of-band write is drift the controller now has to either report or fight:

• kubectl edit / kubectl scale by an operator.
• A CI job that runs kubectl set image for a "quick" hotfix.
• Another controller (an HPA, a mutating webhook) writing a field your manifest also sets.

With self-heal off, Argo CD flags these OutOfSync and leaves them, so your repo now lies about production until someone notices. With self-heal on, the controller reconciles them away on its self-heal timeout, which means the CI kubectl set image hotfix gets clobbered back to git seconds later and the on-call engineer thinks the tool is haunted. Either way the fix is the same: route every change through git. The HPA case is different in kind, a field two owners write, which you resolve by not managing replicas in the manifest and letting the HPA own it, so the controller has nothing to fight.

Wrong "Automated sync means the cluster always matches git." Automated sync detects drift and applies git changes; whether it reverts a live-only change is self-heal, and whether it deletes resources dropped from git is prune. Both default off. Right name the three switches separately (sync, self-heal, prune) and set them per how much hand-tuning your operators legitimately do.

Sync ordering: phases, waves, hooks

Inside a single sync, Argo CD does not fire everything at once. It runs phases in order: PreSync, then Sync, then PostSync (with SyncFail on failure). Within a phase, resources apply by an integer sync wave, lowest number first, default wave 0, negatives allowed for things that must go first. Between waves Argo CD waits (a couple of seconds by default) for the previous wave's resources to report healthy before starting the next.

# a DB migration Job that must finish before the new pods roll
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync            # phase: run before the app
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
---
# an app resource that should apply early in the Sync phase
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"          # lower wave = applied earlier

Hooks are resources tagged with a phase (argocd.argoproj.io/hook); a PreSync Job is the standard place for a schema migration. Their cleanup is a hook-delete-policy: HookSucceeded, HookFailed, or BeforeHookCreation. Flux expresses the same ordering differently: a Kustomization can dependsOn another, and health checks gate what "reconciled" means before dependents proceed. The interview point is that ordering is a first-class, declarative concern here, expressed in the manifests instead of a sleep in a deploy script.

Environment promotion: why branch-per-env hurts

A common wrong turn is one long-lived branch per environment (dev, staging, prod) with promotion as a branch merge. It looks tidy and rots fast. Your shared base manifests exist in all three branches, so any common change collides on merge; environments drift because a fix lands in whichever branch someone merged into; and automated promotion becomes auto-merging branch to branch, which is the fragile part the Argo CD and Flux maintainers explicitly steer people away from.

The pattern that holds up: environments are directories (Kustomize overlays or per-env values files) on one branch. Promotion is a pull request that patches the target environment's overlay, so the whole desired state is visible in one view and the promotion is a reviewable diff. What you promote is an immutable artifact, the image digest or tag you already tested, never a rebuild for prod:

registry: myapp@sha256:9f2c...      # the exact bytes staging validated
staging/kustomization.yaml  -> newTag: 1.8.3
prod/kustomization.yaml     -> newTag: 1.8.2   # promotion = bump this in a PR

Image update automation closes the loop from a new build to a committed change. Flux does it with three resources: an ImageRepository scans the registry for tags, an ImagePolicy selects one (a semver range, say 1.8.x), and an ImageUpdateAutomation checks out a branch, rewrites the tag in the manifests, and commits it back to git. Argo CD Image Updater is the analog. The critical property is that the tool writes the change into git so the reconciler ships it, rather than pushing to the cluster and stepping outside the model.

Rollback semantics and where the command bites

git revert is the rollback. It makes a known-good tree the new HEAD with a real commit (author, time, reason), and reconciliation returns the cluster on the next pass. The audit trail is the point: every deploy and every rollback is a commit you can read.

argocd app rollback (and Flux's suspend-and-pin equivalents) do something narrower and more dangerous to reason about. Argo CD refuses argocd app rollback while automated sync is enabled (it errors rollback cannot be initiated when auto-sync is enabled), so you must argocd app set <app> --sync-policy none first. The rollback then syncs the cluster to a previous revision without changing git or the Application's targetRevision, so the app is now OutOfSync with HEAD. Because targetRevision still points at HEAD, the moment auto-sync and self-heal come back on the controller reconciles the cluster right back to the bad commit and undoes your rollback. The command is for an emergency where you accept that divergence and will follow it with a real revert; it is not a substitute for one.

Wrong "argocd app rollback is the GitOps rollback, it puts us back and keeps git and cluster in sync." It leaves git untouched, so it creates divergence rather than removing it, and self-heal will fight it. Right revert the commit; use the rollback command only as a stopgap you immediately reconcile with a git change.

Secrets and when GitOps is overhead

Secrets are the boundary case for "everything in git." The three live approaches as of mid-2026, all deferring the deep mechanics to secrets-management:

• Sealed Secrets: encrypt with the in-cluster controller's public key (kubeseal), commit the SealedSecret, only that controller can decrypt. Simple, offline, fully in git.
• SOPS: encrypt the values (keys stay readable) against a KMS or age key, commit the file; Flux and others decrypt on apply, just before the Secret hits the cluster.
• External Secrets Operator: git holds only an ExternalSecret reference; the operator reads the real value from Vault or a cloud secret manager and writes the Kubernetes Secret. No secret material in git at all.

Base64 Secrets in the repo are none of these; they are plaintext with an extra step. That is a hard line, not a preference.

GitOps is not free, and a senior says so. You are running a reconciler, structuring a config repo, and standing up secret tooling, all of which is operational surface. For a single small cluster with a handful of services, no multi-environment story, and no audit requirement, a plain push pipeline is less to operate and fails in more obvious ways. GitOps starts paying when drift is a real risk, when you run many clusters or environments, or when "who changed production and when" has to be answerable from history rather than from memory.