10:03 – monitoring detects a sudden increase in API response time.

10:05 – the first alert appears on Slack.

10:08 – one of the engineers opens Grafana.

10:11 – someone asks: "Was there a deployment?"

10:14 – yes, there was!

10:17 – someone starts going through the logs.

10:21 – another engineer joins.

10:24 – finally, someone notices that shortly after the deployment, the number of database connections increased dramatically.

Twenty-one minutes have passed since the first signal.

The team is working, monitoring is working. Alerts are coming in. Everyone is doing exactly what they should be doing,

and the incident is still ongoing.

And somewhere between 10:03 and 10:24, a question worth asking appears:

How many of these activities actually required human involvement?

Probably far fewer than we think.

The first minutes cost the most

When we talk about the effectiveness of Incident Response, we often look at it through the lens of MTTR – Mean Time to Recovery. We want to detect problems earlier, find their causes faster, and restore services to normal operation as quickly as possible.

But when we break down a real incident into individual activities, it turns out that a significant amount of time isn't actually spent solving the problem.

It's spent figuring out what actually happened and where the problem is.

One engineer checks dashboards, another searches through logs.

Someone looks at Kubernetes events. Someone else checks the deployment history.

Then the process of putting everything together begins. Did the latency increase happen before the deployment or after it? CPU looks normal, memory does too. So maybe the database? Or perhaps the problem affects another service that nobody has checked yet?

Going through each of these activities is relatively simple. The problem is that they all have to be done, one after another.

And during an incident, every additional minute means the system continues operating in a state it shouldn't be in.

What if the investigation started without a human?

Let's go back to 10:03.

The API`s response time suddenly jumps from 180 ms to 1.8 seconds.

Normally, we might receive:

ALERT:

API latency above threshold

But what can we actually do with that information? We know something is wrong.

We just don't know why.

Now imagine that the alert automatically triggers an Incident Response workflow.

The system first checks the latest deployments:

10:01 – deployment api-service:v4.18.2

10:03 – latency threshold exceeded

Then it queries Kubernetes:

Pods: healthy

Restarts: 0

CPU: 48%

Memory: 61%

It retrieves the most important application metrics:

HTTP 5xx: +3%

DB connections: +240%

API latency: +900%

And analyzes logs from the period immediately before the alert:

WARN connection pool exhausted

WARN waiting for available DB connection

Two minutes pass.

Before the engineer even manages to open Grafana, the system can present not just a single alert, but the first meaningful context:

The incident started two minutes after the deployment of api-service:v4.18.2.

The application pods are healthy, CPU and memory usage remain stable, while the number of database connections has increased by 240%.

The most likely source of the problem is a change introduced in the latest deployment.

This still isn't the root cause.

But it's a very good place to start the investigation.

Let's automate evidence collection, not just alerts

We have monitoring pretty well automated today. Prometheus detects a threshold breach, Grafana generates an alert, and PagerDuty or Opsgenie notifies the right person. Finally, Slack informs the team.

And very often, this is exactly where the automation ends.

From that point on, a human opens one tool after another and starts collecting data manually.

A little paradoxical.

We've done a great job automating the message:

"Houston, we have a problem."

But we're much worse at automating the answer to the question:

"What happened immediately before the problem?"

And yet the system can automatically collect:

• recent deployments,

• infrastructure changes,

• logs from the affected service,

• key metrics,

• traces,

• Kubernetes events,

• dependency status,

• active feature flags,

• information about similar incidents from the past.

The point isn't to remove the engineer from the process.

The point is to stop them from spending the first fifteen minutes of an incident opening six different tools and a dozen browser tabs.

They probably have too many tabs open already anyway. 😊

The problem starts when there isn't a single "if"

With traditional automation, things are relatively simple. We have a condition and we have a reaction.

CPU exceeds a defined threshold? Trigger a specific action.

A pod stops responding? Restart it.

But a real incident rarely wants to cooperate with our if/else.

High latency may be caused by the application, the database, the network, an external API, an infrastructure change, or a sudden increase in traffic.

Each individual signal tells us very little.

But when we put them together and look at them as a whole, they begin to tell a story:

Deployment: 10:01

Latency increase: 10:03

DB connections: +240%

CPU change: insignificant

Memory change: insignificant

External APIs: healthy

None of these signals gives us the answer.

But it's also difficult to ignore the fact that the deployment happened at 10:01, latency jumped two minutes later, and the number of database connections increased at the same time.

Coincidence? Maybe.

But this is definitely where I would start looking.

And this is where AI makes sense – not because it "knows the answer," but because it can quickly go through data that an engineer would otherwise have to search for across several different places during an incident.

Should AI also fix the problem?

Sometimes, yes.

Let's assume the exact same connection pool issue occurred three months earlier.

The team performed an RCA, found the cause, and documented a safe procedure:

If connection pool exhaustion occurs within five minutes of a deployment,

and the error rate exceeds the defined threshold,

roll back the latest version.

This is no longer guesswork. We have a specific, previously tested procedure.

The system can therefore collect the data itself, check the conditions defined in the runbook and – if all of them are met – perform the rollback.

A moment later, it verifies whether latency and the error rate have returned to normal.

Only then does it report the result.

This can provide enormous value. But it doesn't mean we should allow AI to autonomously perform every possible operation.

Restarting a stateless service? The risk is relatively low.

Rolling back a specific deployment under clearly defined conditions? Also worth considering.

But automatically modifying a production database is out of the question.

Especially if the only argument is that the model decided that "the problem is probably here."

How far should we take this?

We don't have to immediately give automation permission to restart half of production.

At first, it's enough for the system to handle the most tedious part of the work for us: collecting logs, metrics, traces, deployment history, and recent infrastructure changes.

The next step can be analysis. AI compares the data, looks for correlations, and suggests where to look. The engineer still makes the decision.

Only when the scenario is well understood, the conditions are unambiguous, and the runbook has been tested multiple times would I allow automation to take action on its own.

Restart? Rollback? Failover? Sure – but only when we know exactly when the system should do it and, just as importantly, how to undo it.

Because automation without a rollback option can sometimes be just a very fast way to make a mistake.

Let's count again

Let's return to our incident.

With the traditional approach:

10:03 – problem detected

10:08 – investigation begins

10:14 – deployment identified

10:17 – logs analyzed

10:24 – correlation with DB connections discovered

10:31 – decision to roll back

10:37 – service restored

Total: approximately 34 minutes.

Now let's see what changes when some of that work happens automatically. The system collects deployment data, logs, metrics and Kubernetes events, while AI helps connect the first dots.

10:03 – problem detected

10:05 – context collected, hypothesis ready

10:08 – engineer verifies the findings

10:11 – rollback

10:17 – service is operating normally

Total: approximately 14 minutes.

Time saved: approximately 20 minutes.

If a similar incident occurs four times per quarter, we recover almost an hour of system availability.

But the engineer's time isn't the most important factor here.

In a customer-facing system, an additional 20 minutes of degraded service can mean failed transactions, abandoned sessions, SLA breaches, support tickets, and customers wondering whether the competition might be working a little better today.

That's why the value of those 20 minutes can be much greater than the cost of the engineer investigating the incident.

The best incident is still the one that never happens

Automation in Incident Response isn't about creating an autonomous robot that manages production while engineers calmly drink coffee and occasionally check Slack.

Although I have to admit, the coffee part sounds quite reasonable.

It's about something much simpler.

Removing mechanical work at exactly the moment when human attention is needed most.

We don't need a human to compare timestamps across five thousand lines of logs. We need one when the data doesn't provide a clear answer and someone has to decide whether the next move will help or only make things worse.

And that's exactly the kind of division of work that makes the most sense in Incident Response.

So the next time an alert arrives at 10:03, it shouldn't simply say:

"API latency is high."

A much more useful message would be:

"API latency increased two minutes after deployment v4.18.2.

The number of database connections increased by 240%.

Similar symptoms occurred during INC-184.

Recommended first action: verify the latest deployment."

Because knowing that something has broken is useful.

But at 3:00 AM, I'm much more interested in the answer to one question:

Where should I start looking?