Machine Learning

to Test System Resilience

Everything is green.

CPU – 42%. Memory is stable. API response time is below 200 ms. Kubernetes shows that all pods are running properly. No alerts. Nothing particularly interesting in the logs either.

So, a perfect time to break something.

Not in production, of course. At least not today 😊.

We start by adding 150 ms of latency between the application and the database.

Nothing happens.

300 ms – still good.

At 600 ms, response times start to increase. A few timeouts appear, but the application is coping.

So we keep going. We limit the available CPU, add packet loss, and generate a sudden traffic spike at the same time.

At some point, the system stops being resilient and starts simply fighting to survive.

And that is when an interesting question comes up:

Where exactly is that boundary?

Saying that it "works" tells us surprisingly little

Most tests answer a relatively simple question: does the system behave as expected?

We send requests and check the response. We deploy a new version. We run integration tests. We increase traffic. We watch the performance.

With resilience, that is only the beginning.

What happens when the system moves beyond the conditions we designed it for?

An application might handle 500 requests per second without any trouble and start struggling at 700. It might also handle 1,000 requests just fine until database latency increases by another 80 ms.

In practice, a single parameter rarely kills us. The trouble starts when the database slows down, CPU approaches its limit, and another wave of requests arrives at exactly the same time.

Let's assume we want to test just five parameters:

• network latency,

• available CPU,

• memory pressure,

• database response time,

• number of requests.

If we use only five levels for each parameter, we get:

5 × 5 × 5 × 5 × 5 = 3,125 combinations.

Of course, we could test them all.

The only question is who is patient enough to wait for the results – and pay for the environment.

At this point, 3,125 combinations stop being a testing problem and become a selection problem.

Machine Learning starts to make sense

Traditional resilience testing usually starts with scenarios prepared by engineers.

What happens if we lose one pod?

What if the database starts responding more slowly?

How will the application behave if one Availability Zone becomes unavailable?

Tools used in Chaos Engineering allow us to create exactly these situations. We can kill containers, introduce network latency, limit CPU, simulate packet loss, or cut off dependencies.

But someone still has to decide: what do we test next?

This is where Machine Learning (ML) can help. Its job is not to predict failures, but to help us choose the experiment that can give us the most new information.

The first experiments give us the first results. But ultimately, this is not just about collecting a series of outcomes. In the process, we build a dataset in which we can start looking for relationships.

After each test, we record the conditions:

Traffic: 650 req/s
CPU available: 60%
DB latency: 220 ms
Packet loss: 1%
Pods: 4

and the result:

P95 latency: 780 ms
Error rate: 0.8%
Recovery time: 12 s
Result: acceptable

One test gives us one result, but only a series of experiments allows us to start seeing a pattern.

Maybe CPU is not our biggest problem after all.

Perhaps the most dangerous combination is increased database latency, packet loss, and traffic exceeding 800 requests per second.

That tells us something about the limits of the system's resilience, rather than simply giving us another confirmation that the application survived yet another pod restart.

Stop testing what we already know

This is probably the most interesting part of the whole approach.

Let's assume we have already run 200 experiments.

We know that the system behaves predictably at a certain level of available CPU. We also know how it reacts to small increases in network latency.

Do we really need another fifty tests to tell us exactly the same thing?

Probably not.

If, after 200 experiments, we already know that an additional 150 ms of latency changes nothing, test number 201 should not check the same thing again. It makes more sense to run the next experiment in an area where the previous results are no longer obvious.

For example:

Previous tests:

DB latency 100–250 ms + traffic <700 req/s → stable

Unknown area:

DB latency 300–450 ms + traffic 800–1,000 req/s

That is exactly where the next test is worth running.

We are not generating failures just to see whether something breaks.

We are looking for the system's limits.

It is also worth remembering that ML does not make our infrastructure more resilient. It helps us find its weak points.

Find the boundary before production does

Resilience problems do not always appear gradually.

Systems have thresholds.

700 requests/s – everything looks normal.

760 – latency increases slightly.

810 – the connection pool starts filling up.

850 – the retry mechanism generates additional traffic.

880 – the mechanism that was supposed to help us is now making things worse.

And we have found the boundary.

The problem is that production environments are remarkably good at finding these places for us.

Usually at the worst possible moment.

Further experiments show us which combinations of parameters lead to problems.

For example:

Risk of exceeding the SLA

Traffic 600 req/s + DB latency 150 ms → LOW

Traffic 800 req/s + DB latency 250 ms → MEDIUM

Traffic 900 req/s + DB latency 350 ms → HIGH

If the model has never seen how the system behaves under high latency and heavy traffic, it is difficult to expect it to correctly predict what will happen in such a situation. There is no magic here – the model simply uses what it has previously learned from the data.

A failed test is still a good result

There is a certain psychological problem with testing.

Green means good.

Red means bad.

With resilience testing, I would turn that logic around a little.

If all our Chaos Engineering experiments succeed, perhaps we are simply choosing scenarios that are too safe.

A failed resilience test gives us something very valuable.

A boundary.

Maybe the application needs a better circuit breaker. Perhaps the retry policy needs to change. Autoscaling may be reacting too slowly. Or a dependency we considered relatively unimportant turns out to be critical.

For the model, it is another data point. For us, it is something more useful – a specific weak point discovered under controlled conditions rather than at two in the morning after a call from a client.

I definitely prefer that scenario to discovering exactly the same weakness through customer reports.

How many tests do we actually need?

Let's put some numbers behind it.

Let's say we spend 15 minutes on each test. That is enough time to set it up, run it, reset the environment and make sure we can safely move on to the next one.

Testing all 3,125 combinations from the previous example would mean:

3,125 × 15 minutes = approximately 781 hours.

We are unlikely to do that manually.

So let's assume we start with 100 representative experiments. We feed the results back into the model and use them to decide where to go next. Instead of another random batch, we run 50 more experiments around the areas where the previous tests left us with questions.

That gives us:

150 × 15 minutes = 37.5 hours.

37.5 hours is still a lot. But at least we have not spent hundreds more proving for the hundredth time that the system behaves exactly as we expected.

These numbers are only meant to illustrate the scale. In one environment, an experiment might take 30 seconds; in another, half an hour.

The point is not that ML suddenly allows us to eliminate 95% of our tests. The benefit lies somewhere else: we can spend our testing time where we still have the most unknowns.

Resilience is not a single number

At some point, someone will probably ask:

"So what is our resilience score?"

I would be careful with the answer.

A system is not simply "87% resilient."

It might handle the loss of an application instance perfectly well while reacting very badly to a slowdown in an external API. It might handle a database failover without any trouble but stop coping when a retry storm begins.

So a resilience score of 87% would not tell me much. I am more interested in the conditions under which the system starts to fall apart and which combinations of parameters we have not tested at all yet.

Based on the results of subsequent experiments, ML can help us build such a map.

Chaos Engineering allows us to test the system under failure conditions, observability shows us how it reacted, and ML suggests what is worth testing next.

And somewhere in between, we might discover that 350 ms of database latency, 2% packet loss, and 900 requests per second make a particularly bad combination.

Better to find that out during a controlled experiment.

Because sooner or later, a similar scenario may happen in production – except then we will have no control over when it happens or under what conditions.