Detection-to-page versus page-to-fix: the two halves of a Kubernetes incident loop

PagerDuty for Kubernetes: Why Detection-to-Page Is Only Half the Loop

PagerDuty solves getting the right human awake. It does not solve what that human does next, which is where nearly all incident time goes. This covers the split between routing and diagnosis, what actually belongs in each layer, and why replacing your pager is the wrong move.

PagerDuty is very good at the problem it solves: getting the right human awake, fast, with escalation if they do not respond. That problem is solved, and nothing in this article suggests replacing it.

The gap is what happens after the page. Routing an alert takes a few minutes. Working out which container died, why, what the right fix is, applying it, and confirming it held takes most of an hour. No paging tool addresses that stretch, because it was never designed to — and for Kubernetes specifically, that stretch is where nearly all the incident time lives.

This covers what belongs in each layer, what changes when diagnosis arrives with the page, and the honest state of the integrations.

What a paging tool is actually for

PagerDuty, Opsgenie and their peers answer a specific set of questions well:

  • Who is on call for this service right now?
  • Did they acknowledge, and if not, who is next in the escalation policy?
  • How do we suppress this during a maintenance window?
  • What is our acknowledgement time against the SLA we promised?

Those are scheduling and routing problems. They are largely independent of what the alert says — the same escalation policy works whether the payload is a Kubernetes OOMKill or a payment gateway timeout. That generality is the strength, and it is also the reason a pager tool cannot tell you why your pod died.

Where the time actually goes

Incident timeline showing routing takes about four minutes while evidence gathering, diagnosis, fix and verification take about forty-eight

Take a routine crash loop. Alert fires at 02:04, on-call acknowledges at 02:09, root cause identified at 02:23, fix merged at 02:38, verified stable at 02:51.

See what your next page could arrive with.

Connect a cluster read-only in about 5 minutes and get the live issue list with causes already attached. No write access, nothing changes.

Connect read-only →

Routing consumed five minutes of a forty-seven minute incident. Everything else was a human reading logs, checking recent deploys, comparing the current memory limit against actual usage, and deciding what number to put in the patch.

Optimising the five minutes has a small ceiling. Even instantaneous paging saves four minutes on an incident where forty-two were spent elsewhere. This is the arithmetic behind the whole argument — and it is also why "we already have PagerDuty" is not an answer to "how do you shorten incidents."

What changes when diagnosis arrives with the page

Comparison of a bare Kubernetes alert against the same alert carrying exit code, container, limit, peak and a proposed fix

A standard KubePodCrashLooping alert tells you a pod is crash looping in a namespace. Everything else you find out yourself, starting from a laptop at 2am.

The same alert, with an agent having already read the cluster, carries the exit code and reason, which container terminated (not necessarily the first one in the pod), the current limit against observed peak, the restart count, a proposed value with the basis it was derived from, and a pull request already open for review.

The on-call engineer's first action shifts from investigate to review a proposal. That is a different cognitive task, and it is dramatically easier at 2am. The page still arrives through PagerDuty. Only the payload changed.

This is also why the two layers compose rather than compete. Grouping matters here — an agent that raises one incident per root cause instead of forty pod alerts means your escalation policy fires once, which is a direct improvement to the paging layer's output. We covered that in alert fatigue in Kubernetes.

Where the responsibilities split

CapabilityPaging toolSRE agent
On-call schedules and rotationsYesNo
Escalation policiesYesNo
Maintenance windows and suppressionYesNo
Acknowledgement SLAs and reportingYesNo
Multi-service, non-Kubernetes alertsYesNo
Reading pod specs, events and container stateNoYes
Determining root cause from evidenceNoYes
Proposing a specific patch with a derived valueNoYes
Verifying the fix held afterwardsNoYes

There is no row where they overlap. That is the actual finding — these are adjacent tools, not competing ones, and any vendor pitching one as a replacement for the other is describing a product you should look at more carefully.

The honest integration position

Worth stating plainly, because this is where vendor content usually gets vague.

Atmosly Astra does not integrate with PagerDuty or Opsgenie today. Notifications reach Slack, Microsoft Teams, email and in-app, with a per-organisation severity allowlist. PagerDuty and Opsgenie webhooks are roadmap, not shipped.

What that means practically: today the agent's diagnosis lands in Slack while your page lands in PagerDuty, and the person on call correlates them. That is a real seam, and it is worth knowing before you plan around it.

Two more limits in the same spirit. Slack integration is one-way — a card with a deep link, not interactive buttons. You cannot approve or roll back a fix from Slack. And there is no incident timeline export, so postmortem assembly stays manual; the Kubernetes postmortem template covers doing that by hand.

What to keep, what to add

A decision framework rather than a recommendation to switch.

Keep your paging tool if you have on-call rotations across more than one service, you page for anything outside Kubernetes, you have acknowledgement SLAs to report against, or you need maintenance-window suppression. All four are true for most teams past about fifteen engineers, and none of them are things an SRE agent does.

Add an agent layer if most of your incident time is spent after the page rather than before it, the same failure classes keep recurring, your on-call engineers are not the people who wrote the service, or you cannot currently answer "how long does diagnosis take us" with a number.

Do not add one if your incidents are mostly application logic rather than cluster state. An agent reading Kubernetes state can tell you a pod restarted and which container died; it cannot tell you your payment service deadlocks under a particular concurrency pattern. That boundary is real and it is roughly half of production incidents.

How to evaluate the seam

If you run both layers, these are the questions that matter:

  1. What does the on-call engineer see first? If the page and the diagnosis live in different tools, someone is doing correlation at 2am. Measure how long that takes before assuming it is trivial.
  2. Does grouping upstream reduce pages downstream? One incident per root cause should mean one page, not forty. This is the clearest measurable win from adding an agent in front of a pager.
  3. Can the agent's proposal be reviewed without leaving the alert? A deep link is acceptable. Re-deriving context from scratch is not.
  4. What happens when the agent is wrong? The escalation path must still work. An agent that suppresses a page because it thinks it has handled something is a worse failure than no agent at all.

That last one is the trap. The value is in enriching what gets paged, not in deciding what does not get paged. Any tool offering to reduce your page volume by resolving things silently should be asked exactly what it does when its own fix fails — AI SRE for Kubernetes covers what verification and rollback should look like.

The short version

Detection-to-page is a solved problem and PagerDuty solves it well. Detection-to-fix is not solved, it is where roughly ninety percent of incident minutes go, and it is a different category of tool.

Run both. Judge the agent layer on whether the page arrives with a cause attached and whether the same incident stops recurring — not on whether it reduced the number of pages, which is the metric easiest to game and the one that hides the most.

You can see what arrives with the page against your own cluster without granting write access — Atmosly Astra runs read-only, and you can connect a cluster free.

Frequently Asked Questions

Does an AI SRE agent replace PagerDuty?
No, and treating it as a replacement is a mistake. A paging tool answers scheduling and routing questions — who is on call, did they acknowledge, who is next in the escalation policy, how do we suppress during maintenance. An SRE agent answers diagnostic questions about cluster state. There is no capability where the two genuinely overlap, so they compose rather than compete.
Does Atmosly integrate with PagerDuty?
Not today. Astra sends notifications to Slack, Microsoft Teams, email and in-app, with a per-organisation severity allowlist. PagerDuty and Opsgenie webhooks are on the roadmap but are not shipped. In practice this means the diagnosis lands in Slack while the page lands in PagerDuty, and the on-call engineer correlates them — a real seam worth planning around.
Why is optimising alert routing not enough to reduce incident time?
Because routing is a small share of the total. In a typical crash loop — alert at 02:04, acknowledged 02:09, cause found 02:23, fix merged 02:38, verified 02:51 — routing consumed five minutes of a forty-seven minute incident. Even instantaneous paging saves four minutes on an incident where forty-two were spent gathering evidence, diagnosing and verifying.
What should a Kubernetes alert contain to be useful at 3am?
Beyond the pod and namespace: the exit code and termination reason, which container actually died rather than assuming the first one, the current resource limit against observed peak, the restart count, and a proposed fix with the basis it was derived from. That shifts the on-call engineer's first action from investigating to reviewing a proposal, which is a far easier task under pressure.
Can I approve or roll back a fix from Slack?
Not in Atmosly today. Slack integration is one-way — a notification card with a deep link back to the platform, not interactive buttons. Approving, applying or rolling back a remediation requires opening the platform. Worth confirming this specifically with any vendor, since ChatOps depth varies a lot and is often described more generously than it works.
Should adding an AI SRE agent reduce my page volume?
Only through grouping, and that is the right reason. If forty pod alerts from one root cause become one incident, your escalation policy fires once instead of forty times. Be cautious of any tool that reduces page volume by silently resolving issues — ask exactly what happens when its own fix fails, because an agent that suppresses a page for something it did not actually handle is worse than no agent.
When should a team keep its paging tool rather than consolidating?
Keep it if you have on-call rotations across more than one service, page for anything outside Kubernetes, report against acknowledgement SLAs, or need maintenance-window suppression. All four are typical past roughly fifteen engineers, and none of them are things an SRE agent does. Consolidation here removes capability rather than reducing tool sprawl.
When is an AI SRE agent not worth adding alongside PagerDuty?
When your incidents are mostly application logic rather than cluster state. An agent reading Kubernetes state can report that a pod restarted and which container died, but it cannot diagnose a deadlock in your payment service under a specific concurrency pattern. That boundary covers roughly half of production incidents and no tool in this category crosses it.
What is the difference between MTTA and MTTR in this context?
MTTA measures alert to human acknowledgement, which is the interval a paging tool owns and optimises. MTTR measures from cause known to verified resolution, which is where an SRE agent operates. Teams reporting a single blended number cannot tell which half improved, which makes it impossible to judge whether the paging layer or the diagnosis layer produced a change.
How do I evaluate the seam between a pager and an SRE agent?
Four questions: what does the on-call engineer see first, and how long does correlating two tools take at 2am; does upstream grouping actually reduce downstream pages; can the agent's proposal be reviewed without re-deriving context from scratch; and does the escalation path still work when the agent is wrong. The last one matters most — the value is in enriching what gets paged, not deciding what does not.
Does an SRE agent handle on-call schedules or escalation policies?
No. Schedules, rotations, escalation policies, maintenance windows and acknowledgement SLA reporting are all paging-tool responsibilities, and an SRE agent implements none of them. This is why the two layers are complementary — removing the paging tool to consolidate would leave you without any of that scheduling capability.
What should I measure to judge whether the agent layer is working?
Two things: whether the page now arrives with a root cause attached, and whether the same incident class stops recurring within seven days. Do not judge it on reduced page volume — that metric is the easiest to game, since it improves both when incidents genuinely fall and when a tool quietly stops telling you about them.