A pod starts failing at 2am. The reflex fix is a quick kubectl patch or kubectl scale against production. On a cluster managed by GitOps, that fix has a short life: the reconciler compares the live state to the Git repository, sees your change is not in the repo, and reverts it on the next sync. The incident comes back. This is why GitOps remediation exists — a fix only counts when it lands in the source of truth the cluster reconciles to.
GitOps remediation is the practice of resolving Kubernetes incidents by changing the Git repository — ideally through an automated, reviewable pull request — rather than mutating the live cluster directly. Done well, it turns "someone SSHed in and patched prod" into "an incident opened a PR, a human approved it, and the controller rolled it out." This guide explains the pattern, why a direct patch is not remediation on a GitOps cluster, how the alert-to-PR flow works, and how to keep automated remediation safe.
What GitOps remediation actually means
GitOps has one governing rule: the Git repository is the single source of truth, and a controller such as Argo CD or Flux continuously reconciles the cluster to match it. That principle — declarative, versioned, continuously reconciled — is the foundation covered in our GitOps for Kubernetes implementation guide.
Remediation is what you do when something breaks. Combine the two and the definition falls out: GitOps remediation is fixing an incident by committing the corrective change to Git, so the fix flows through the same reconcile loop as every other deployment. The change is declarative, reviewed, versioned, and auditable — not an out-of-band mutation that drifts away.
Why a direct kubectl patch is not remediation
On a GitOps-managed cluster, a live edit and the repo are now in conflict, and the controller always wins.
A direct patch is reverted at the next sync; a merged PR becomes the desired state and sticks.
- Direct patch: you edit the live object, the controller reconciles, your change does not match the repo, and it is reverted. The fix is gone and you have quietly introduced drift.
- Pull request: the fix is committed to the manifest, reviewed, and merged. The repo now describes the desired state, the controller reconciles to it, and the change is durable and audited.
This is the core reason "automated remediation" and "GitOps" have to be designed together. An auto-remediation bot that issues live patches on a GitOps cluster will fight the reconciler forever. The durable answer is always to land the change in Git.
The alert-to-PR pattern, step by step
The modern approach — increasingly called automated Kubernetes remediation — closes the loop from a detected incident to a merged pull request. The pattern generalizes across tools:
The alert-to-PR loop: detect, find the owning application, patch the manifest, open a reviewable PR, a human merges, then verify.
- Detect the incident. A failing workload — OOMKilled, CrashLoopBackOff, ImagePullBackOff, a NotReady node — is identified with enough evidence to reason about the cause.
- Find the owning application. Map the failing workload to the Argo CD Application (or Flux Kustomization) that manages it, and to the manifest in the backing repository.
- Patch the manifest in Git. Apply the corrective change to the YAML — a resource limit, a probe, a replica count — preserving the file's existing formatting so the diff is minimal.
- Open a pull request. Raise a PR with a clear diff, the root cause, and a rollback note, against GitHub, GitLab, or Bitbucket.
- A human merges. The change flows through your existing review and branch-protection rules. Nothing reaches production without approval.
- Verify. After the controller reconciles the merged change, confirm the workload is healthy — and if not, revert the commit.
Keeping automated remediation safe
Autonomy without guardrails is how "self-healing" becomes an outage amplifier. The patterns that make automated remediation trustworthy are well established, and every credible implementation — including agentic operators that use an LLM to draft the fix, as described in this DZone write-up — leans on them:
- Human-in-the-loop by default. A pull request that a person reviews and merges beats a silent apply. Reviewability is the feature, not a limitation.
- Read-only detection. The component that watches the cluster and proposes fixes should need read-only access. Write access, if any, is a separate, explicit grant.
- Policy checks in the pipeline. Run OPA/Gatekeeper or Kyverno and CI validation on the PR before it can merge, so a proposed change still has to pass your rules.
- Branch protection on production paths. Require review for any change to production namespaces; never auto-merge into prod.
- Reversibility. Every automated change should carry a rollback — with GitOps, that is simply reverting the commit.
Meeting your repository where it is
GitOps remediation is only useful if it works with the Git provider and controller you already run. In practice that means opening PRs against GitHub, GitLab, or Bitbucket through existing credentials, and discovering ownership from your GitOps tool rather than asking engineers to wire up mappings by hand. If you are still choosing a controller, our Argo CD vs Flux comparison covers the trade-offs that affect how remediation plugs in.
How Atmosly approaches GitOps remediation
Atmosly's AI SRE agent implements this pattern end to end: it detects a failing workload, groups the noise into one incident, identifies the owning Argo CD Application, patches the manifest in your repo, and opens a pull request that you review and merge. A human always merges the PR, and detection runs read-only — the guardrails above are the design, not an add-on. For a single incident walked through in detail, from an OOMKill alert to a merged fix, see the anatomy of an AI SRE fix. It sits alongside the other tools in the space, compared in our guide to the best AI SRE tools in 2026.
What to auto-remediate — and what to leave to a human
Not every incident should be resolved the same way. A useful rule: automate the changes that are bounded, reversible, and evidence-driven; escalate the ones that need architectural judgment. In practice, the "safe to draft as a PR" set is narrow and high-value:
- Resource requests and limits. An OOMKilled workload with a clear working-set signal is the canonical case — raise the memory limit, open a PR, done.
- Probes. A liveness or readiness probe that is too aggressive and causes needless restarts is a small, safe manifest change.
- Replica counts. Scaling a Deployment that is under-provisioned for its load, within sane bounds.
By contrast, anything that touches data integrity, security posture, network policy, or application logic should stop at a recommendation and route to a human. The goal of Kubernetes auto remediation is not to remove engineers from the loop; it is to hand them a ready-to-review change for the repetitive failures and reserve their attention for the genuinely novel ones. Pairing this with incident grouping — so ten crash-looping replicas become one incident, as covered in our guide to fixing Kubernetes alert fatigue — is what makes the workflow scale.
Best practices checklist
- Never remediate a GitOps workload with a live patch — commit to the repo the controller reconciles to.
- Automate the boring 80% (resource limits, probes, replicas) as PRs; keep humans on the merge.
- Gate every PR with policy checks and CI, and protect production branches.
- Keep detection read-only and grant write access narrowly and explicitly.
- Make rollback trivial — one revert — and verify health after reconcile.
Key takeaways
- GitOps remediation means fixing incidents in Git, so the corrective change flows through the same reconcile loop as every deployment.
- A direct
kubectlpatch is not remediation on a GitOps cluster — the controller reverts it and you get drift. - The durable pattern is alert-to-PR: detect, find the owning app, patch the manifest, open a reviewable PR, merge, verify.
- Safety comes from human-in-the-loop, read-only detection, policy checks, and branch protection — autonomy without these amplifies outages.
- Automated GitOps remediation removes the toil, not the judgment: the agent drafts and opens the fix; a human still approves it.
Want to see an incident on your own cluster arrive as a reviewable GitOps pull request instead of a 2am patch? Connect Atmosly read-only and run a free audit in about five minutes. Start with Atmosly.
