Atmosly
Login Connect cluster →
Four tiers of AI SRE access: read-only, pull request, approval gate, and direct apply
By Ankush Madaan August 5, 2026 AI SRE

AI SRE Security: Safe AI Automation for Production Kubernetes

Every AI SRE evaluation reaches the same objection: you want an AI to have production access? This is the honest answer — what to grant, what to refuse, the RBAC to demand instead of cluster-admin, and the audit record that gets it past a security review.

ai-governanceai-sreaudit-trailincident-responsekubernetes-securityleast-privilegerbacsecure-automation

Every AI SRE evaluation reaches the same meeting, and the same sentence: you want an AI to have production access?

It is the correct question and it deserves a better answer than reassurance. The short version: an AI SRE agent should need no write access at all to deliver most of its value, the write access it eventually needs is a short list of specific RBAC verbs rather than cluster-admin, and every mutation must leave a record that answers who approved it and how to undo it. If a vendor cannot hand you the exact Role YAML, you have your answer.

This covers what to grant at each stage, what to refuse, the permission model to demand, and the audit record that gets an agent past a security review. For how the underlying loop works, see the AI SRE for Kubernetes guide.

Does an AI SRE agent need production access?

The objection assumes a single component with production credentials. That assumption is worth testing, because a well-built agent splits into two.

Detection, correlation, diagnosis and proposal need read access only. Reading pod specs, events, logs and node conditions requires no write verb anywhere. An agent doing that work is a monitoring consumer with a language model attached, and it carries roughly the risk profile of your existing observability stack.

Only delivery and rollback need to mutate. That is one stage out of seven.

Atmosly ships this as two distinct agent flavours — a read-only debug agent and a separate mutation-capable ops agent — so the split is enforced by which workload you deploy, not by a configuration flag someone can flip. A team can run detection indefinitely without the second one existing in the cluster.

The practical consequence: your first evaluation should not involve granting write access at all. If a vendor requires it to demonstrate value, that is information about the product.

What to demand instead of cluster-admin

Comparison of a cluster-admin grant against the scoped RBAC verbs a detection agent and a remediation agent actually need

Cluster-admin is the default ask because it is the path of least resistance for a vendor shipping an install chart. It is almost never the requirement.

Two identities, two permission sets. Here is what each one actually needs.

The detection identity

A read-only agent needs get, list and watch on the resources it inspects, plus pods/log. Nothing else:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sre-agent-read
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "events", "nodes",
                "persistentvolumeclaims", "configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["get", "list", "watch"]

Note what is absent. No secrets — an agent that reads pod specs sees secret references, which is enough to diagnose a missing-secret failure without reading values. If a vendor wants get on secrets, ask precisely which diagnosis requires it.

Try it without granting write access.

Connect a cluster read-only in about 5 minutes and see the live issue list. No mutation verbs, nothing changes in your cluster.

Connect read-only →

The write side is a separate identity, and a shorter list.

The remediation identity

A separate ServiceAccount, with patch and update on the workload kinds it can actually fix, scoped to named namespaces via RoleBindings rather than a ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sre-agent-remediate
  namespace: payments
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets"]
    verbs: ["get", "patch"]

Two identities rather than one is the load-bearing decision. A compromised detection agent leaks information; it cannot mutate anything. Collapse them into a single ServiceAccount and you have traded that property away for one less YAML file. Our Kubernetes RBAC guide covers the binding mechanics.

Verify the grant rather than trusting the chart

Install charts drift from their documentation. Check what was actually created:

# What can the agent's identity do?
kubectl auth can-i --list \
  --as=system:serviceaccount:atmosly-system:sre-agent-read

# Specifically, can it do the thing it should not?
kubectl auth can-i delete pods \
  --as=system:serviceaccount:atmosly-system:sre-agent-read -A
kubectl auth can-i get secrets \
  --as=system:serviceaccount:atmosly-system:sre-agent-read -A

Run this after install and again after every chart upgrade. A permission added in a minor version bump is the kind of change nobody reads the release notes for.

Git as the safest write path

The strongest security property in this category is not a permission model. It is delivering the change as a pull request instead of an API call.

A fix that arrives as a diff inherits every control you already have: branch protection, required reviewers, CODEOWNERS, signed commits, CI checks. The agent needs no cluster write permission for this path — it needs repository access, which your existing review process already governs. GitOps remediation for Kubernetes covers the mechanics.

Two caveats worth stating plainly.

First, repository write access is not nothing. Scope the agent's token to the specific repositories holding manifests, never to an org-wide token, and require review on the branches it targets.

Second, Atmosly's GitOps detection is ArgoCD-only. If Flux or a Helm operator manages your workloads, the agent cannot confirm GitOps ownership, the sync-pause protection does not apply, and a direct patch may be reverted on the next reconcile. Know which path you are actually on before choosing it as your safety mechanism.

The approval gate

For direct applies, the control that matters is a human approving a specific change with their identity attached.

Atmosly parks direct applies as pending_approval rather than executing them, and an approver holding the approve_remediation permission approves or rejects. The approver's identity is recorded on the audit row at the moment of approval.

Three things to check in any implementation:

  • Can requester and approver be different people? An approval an agent grants itself is theatre.
  • Does the approver see the actual change? Approving "apply the recommended fix" is not approving a specific patch to a specific container.
  • Can the value be edited before approval? If the proposed memory limit looks wrong, an approver who can only accept or reject will accept.

One honest limitation: Atmosly's approval gate is a global setting today, not per-namespace or per-environment. There is no policy engine enforcing blast-radius caps, namespace allow-lists or per-environment autonomy tiers. If your security review requires enforced autonomy limits, that gap is the thing to raise.

What the audit trail must record

Six questions an audit record must answer for every automated remediation, including rollback spec and verification verdict

A log line reading remediation applied answers none of the questions an auditor will ask. A usable record captures six things per change: who requested it, who approved it, the exact patch including object, container, old value and new, the rollback specification captured before the mutation, whether an operator overrode the suggested value, and what the post-apply verification returned.

That fifth field is the underrated one. When an operator changes 1152Mi to 900Mi before approving, the record should hold both numbers. Six months later, a pattern of operators consistently reducing proposed values tells you the sizing logic is miscalibrated — a signal you cannot recover if only the applied value was stored.

Ask any vendor to show you one real record, populated, from a real incident. Reading an actual row tells you more than any architecture diagram.

Kubernetes' own audit logging is the independent check. Every agent mutation appears there attributed to its ServiceAccount, which lets you verify the agent's own record is complete rather than taking it on trust.

AI SRE threat model: four risks worth planning for

Four risks specific to this category, in the order they should worry you.

Confidently wrong diagnosis. The realistic failure is not a hijacked agent; it is a correct-looking fix for a misdiagnosed problem. Mitigation is verification and reversibility, not access control. An agent that re-checks after applying and reverts on regression contains this. One that records success and moves on does not.

Scope creep through upgrades. The chart you audited in March is not the chart running in September. Pin versions and re-run kubectl auth can-i --list on every upgrade.

Sensitive data in prompts. Diagnosis sends logs and pod specs to a model. Logs contain whatever your application prints, including tokens developers did not intend to log. Ask where inference runs, what is retained, and whether anything is used for training. This is a data-residency question as much as a security one.

Prompt injection through cluster content. An agent reading container logs is reading attacker-influenceable text. A log line crafted to read as an instruction is a real class of attack against any log-reading LLM system. The structural defence is that the agent emits a typed remediation from a fixed set of primitives rather than free-form commands — a proposal that is not one of the supported primitives cannot execute, whatever the log said.

A rollout that passes review

  1. Read-only, non-production, two weeks. No write verbs. Verify with kubectl auth can-i --list rather than the vendor's word.
  2. Grade the diagnoses. For each incident, would you have reached the same conclusion? This is your evidence base and it costs nothing.
  3. Enable the pull request path in staging. Still no cluster write permission — repository access only, on scoped repositories.
  4. Production, read-only plus PRs. Most of the value, none of the direct mutation.
  5. Direct apply, narrowly. One namespace, reversible primitives only, approval gate on, separate ServiceAccount. Expand on track record, not on schedule.

Teams that skip to step five and hit one bad fix disable everything and do not return. The evidence gathered in steps one and two is what makes the later steps defensible to the people who have to sign off.

The honest summary: an AI SRE agent does not need production write access to be useful, it should never need cluster-admin, and the controls that matter are the ones you can verify yourself with kubectl rather than the ones described in a datasheet. Runbook automation covers how far to let remediation go once the access model is settled.

You can evaluate the whole diagnosis path with no write permission at all — Atmosly Astra ships a read-only agent for exactly this, and you can connect a cluster free.

Frequently Asked Questions

Does an AI SRE agent need cluster-admin access?
No, and a vendor asking for it is asking for convenience rather than a requirement. A detection agent needs get, list and watch on pods, events, nodes and workload kinds, plus pods/log. A remediation agent needs patch and update on the specific workload kinds it can fix, scoped to named namespaces through RoleBindings rather than a ClusterRoleBinding. If a vendor cannot hand you the exact Role YAML, treat that as the answer.
Can I run an AI SRE agent without giving it any write access?
Yes, and it should be your first evaluation step. Detection, correlation, diagnosis and proposal all work with read-only access — that is six of the seven loop stages. Atmosly ships this as a separate read-only debug agent, so the split is enforced by which workload you deploy rather than by a configuration flag. If a vendor needs write access to demonstrate value, that tells you something about the product.
Why should the detection agent and the remediation agent be separate identities?
Because a compromised detection agent should leak information, not mutate your cluster. Two ServiceAccounts with different permission sets preserve that property. Collapsing them into one identity trades it away for one less YAML file. It also gives you clean attribution in Kubernetes audit logs — you can tell reads from writes by which account performed them.
Does an AI SRE agent need access to Kubernetes secrets?
It should not. An agent reading pod specs sees secret references, which is enough to diagnose a missing-secret or misconfigured-reference failure without reading any values. If a vendor requests get on secrets, ask which specific diagnosis requires the value rather than the reference. In most cases the answer reveals the permission set was written for convenience.
How do I verify what permissions an AI agent actually has?
Do not trust the install chart's documentation. Run kubectl auth can-i --list --as=system:serviceaccount:NAMESPACE:AGENT-SA to enumerate the real grant, then test specific negatives such as kubectl auth can-i delete pods and kubectl auth can-i get secrets for that account. Re-run this after every chart upgrade, because a permission added in a minor version bump is exactly the change nobody reads release notes for.
Is delivering fixes as pull requests safer than direct patches?
Yes, and it is the strongest security property available in this category. A fix arriving as a diff inherits branch protection, required reviewers, CODEOWNERS, signed commits and CI checks. The agent needs no cluster write permission at all on this path. The trade-off is repository access, so scope the token to the specific manifest repositories rather than issuing an org-wide token.
What should an audit record capture for an automated fix?
Six things: who requested it, who approved it, the exact patch including object, container, old value and new value, the rollback specification captured before the mutation, whether an operator overrode the suggested value, and what the post-apply verification returned. A log line saying 'remediation applied' answers none of these. Ask a vendor to show you one real populated record from an actual incident.
Why does it matter whether an operator overrode the suggested value?
Because it is your calibration signal. If the record stores both the proposed value and the applied one, a pattern of operators consistently reducing proposed limits tells you the sizing logic is miscalibrated. That information is unrecoverable if only the applied value was stored, and it is the data you need before widening autonomy.
Can prompt injection through container logs make an agent run malicious commands?
It is a real risk class, since an agent reading container logs is reading attacker-influenceable text. The structural defence is that a well-built agent emits a typed remediation chosen from a fixed set of primitives rather than free-form shell or kubectl commands. A proposal that is not one of the supported primitives cannot execute regardless of what a log line said. Ask vendors whether their action space is constrained this way.
What is the biggest security risk with AI SRE, realistically?
Not a hijacked agent — a confidently wrong diagnosis producing a correct-looking fix for a misdiagnosed problem. Access control does not mitigate this; verification and reversibility do. An agent that re-checks the workload after applying and reverts automatically when a new problem appears contains the damage. One that records success and moves on leaves you to discover it later.
Where does the AI inference run, and what happens to my logs?
Ask this explicitly, because diagnosis sends container logs and pod specs to a model, and logs contain whatever your application prints — including tokens developers did not intend to log. Establish where inference runs, what is retained and for how long, and whether any content is used for training. It is a data-residency question as much as a security one, and it belongs in the evaluation rather than the contract review.
How should we roll out an AI SRE agent so it passes a security review?
Read-only in non-production for two weeks, verified with kubectl auth can-i rather than the vendor's word. Grade every diagnosis against your own conclusion to build an evidence base. Enable the pull request path in staging, which needs repository access rather than cluster write. Move to production with read-only plus PRs. Only then enable direct apply, in one namespace, reversible primitives only, with the approval gate on.