Drift Detection and Self-Healing Environments
The article explores how AI-driven Policy-as-Code can detect configuration drift and automatically respond when cloud infrastructure no longer matches its defined state. It shows how combining IaC, policies, and AI can create more reliable, self-healing environments while reducing manual work for DevOps engineers.

Tomasz Olszowy
•
8
min.
Nothing particularly unusual happened over the weekend. Monday morning brought no unexpected surprises either. No major deployments, no emergency work, no suspicious alerts.
At least, that was what everyone thought.
According to the Terraform configuration in the repository, one of the production databases should be accessible exclusively from the application subnet. The actual cloud environment, however, tells a different story. After a short investigation, it turns out that someone added another network rule directly through the cloud provider's console on Friday afternoon.
Quite possibly, it was used to troubleshoot an issue, and with the weekend already on their mind, someone simply forgot to remove it afterwards.
The facts are simple: Terraform knows nothing about this rule. There is no mention of it in the documentation either, but unfortunately, production has already felt its presence.
This, or something very similar, is how many infrastructure problems begin — often in surprisingly mundane ways.
Infrastructure as Code Does Not Mean Infrastructure Never Changes
IaC has completely changed the way we manage cloud environments. With Terraform, OpenTofu, Pulumi, or CloudFormation, we can describe, review, and version infrastructure much like application code.
The principle is simple: code defines the infrastructure. Unfortunately, there are exceptions to every rule — and they happen more often than we might like.
During an incident, one engineer manually changes a security group rule, while someone else increases the size of a virtual machine. Another person disables a monitoring rule while troubleshooting an issue and forgets to enable it again.
None of these actions necessarily results from bad engineering practices. Sometimes production simply needs to be fixed immediately, without waiting for someone to prepare the perfect pull request.
And in situations like these, the problem appears later.
What is actually running in the cloud no longer matches what has been defined in Git.
We call this difference configuration drift.
A simple example:
resource "aws_security_group_rule" "database" {
type = "ingress"
from_port = 5432
to_port = 5432
protocol = "tcp"
cidr_blocks = ["10.0.10.0/24"]
}
According to Terraform, PostgreSQL accepts traffic exclusively from the internal network.
But someone manually adds:
0.0.0.0/0 -> TCP/5432
The code is still correct.
The deployed infrastructure is not.
At this point, the key question is no longer whether we use Infrastructure as Code, but whether we can actually trust it.
Policy-as-Code Becomes a Protective Guardrail
This is exactly where Policy-as-Code becomes useful.
Instead of documenting security and infrastructure rules in yet another PDF that — let's be honest — nobody is going to read during an incident, we describe them in a form that systems can automatically interpret and enforce.
Policies can specify, for example, that:
databases must not be publicly accessible,
production data must be encrypted,
resources must have the required tags,
administrative ports must not be exposed to the Internet,
production environments must not use unapproved instance types,
backups must be enabled for critical databases.
Tools such as Open Policy Agent, Conftest, or HashiCorp Sentinel can automatically verify infrastructure against these rules.
And that alone is already extremely valuable.
We are no longer relying on a simple instruction:
"Please don't expose the database to the Internet."
Instead, we make the platform enforce that rule itself.
There is still one limitation, however.
Traditional Policy-as-Code is very good at answering the question:
Does this configuration comply with a specific rule?
It is much less capable when the questions become more complex:
Why did the configuration change, how much risk does it create, and what should we do about it right now?
This is where AI can add another layer.
Not All Drift Is the Same
Imagine that monitoring detects three changes.
Change A: someone added a missing cost-center tag.
Change B: a virtual machine in the development environment was changed from t3.medium to t3.large.
Change C: a production database became publicly accessible.
Technically, all three cases qualify as configuration drift.
Operationally, however, they are completely different.
A traditional mechanism will detect and report the differences. An AI-powered system can additionally analyze their context.
It can take into account the change history, environment type, resource criticality, violated policies, previous incidents, and dependencies between individual resources.
Instead of generating three identical alerts, it could classify them like this:
Drift A
Risk: LOW
Action: synchronize the state during the next deployment
Drift B
Risk: MEDIUM
Action: verify the change with the infrastructure owner
Drift C
Risk: CRITICAL
Action: immediately restore the approved network policy
This completely changes the role of drift detection. The system is no longer merely looking for differences — it is trying to understand which of them actually matter.
From Detection to Self-Healing
Detecting problems is extremely useful, but discovering one at an abstract 2:43 a.m. and then waiting until 8:30 for someone to read the generated alert is difficult to call full automation.
With properly defined policies, we can move to the next step: self-healing infrastructure.
But let's return to our database example.
The platform detects that port 5432 has been publicly exposed, while Policy-as-Code confirms a violation of the production environment's security policy. At the same time, the AI layer analyzes the context and classifies the change as high risk.
The remediation process can then automatically:
restore the approved security group configuration,
create an audit event,
record the detected difference,
identify the source of the change,
notify the responsible team.
Instead of:
detect → alert → wait → analyze → fix
we get:
detect → assess → fix → explain
That is a significant difference.
There is, however, one important rule.
AI cannot have unlimited permissions to "fix" production.
That would be an exceptionally creative way to create a new incident while resolving the previous one.
Self-healing should operate within strictly defined, low-risk boundaries. Well-understood violations can be remediated automatically. Medium-risk cases don't necessarily need to be handled automatically. In many situations, opening a pull request or asking someone to approve the proposed change is the safer option. And when the situation is unclear, unusually complex, or potentially disruptive, the decision should remain with an engineer.
That distinction matters. AI can help determine what is happening and suggest what to do next, but it shouldn't be the component that decides what it is allowed to do. That boundary still belongs to policy.
Does This Actually Save Time?
The benefit becomes easier to see with a fairly ordinary example.
Imagine a team responsible for around 120 cloud resources spread across development, staging, and production. They review the infrastructure twice a month to check whether it still matches their security and compliance requirements.
If one of those reviews takes roughly three hours, that is already about six hours of engineering time every month spent mostly checking configuration and looking for inconsistencies.
And that is when everything is more or less fine.
As soon as drift appears, someone has to investigate it: compare the expected state with what is actually running, work out when the change happened, decide whether it was intentional, and determine what should be done about it. Even a handful of these cases can easily add another few hours to the monthly workload.
If four such cases per month require approximately 45 minutes each:
4 × 45 minutes = another 3 hours
That gives us approximately 9 hours per month spent verifying infrastructure.
With automated policy evaluation and drift detection, engineers no longer need to manually compare most resources. If the remaining verification and exception handling takes approximately two hours per month, the theoretical saving is:
9 h – 2 h = 7 hours per month
Of course, this is an example, not a universal benchmark. The actual result depends on the size of the infrastructure, frequency of changes, and maturity of the automation.
What matters, however, is the direction.
Engineers spend less time trying to answer the question "Who changed this?" and more time improving the platform.
There Is Something Else That Is Much Harder to Measure
The concept of the "biggest benefit" is relative, and it does not necessarily have to be the time saved.
It can also — or perhaps primarily — be trust.
Developers know that an emergency manual change introduced into the production environment may remain there for months, affecting how that environment operates the entire time.
Security teams know that critical policies are being continuously verified rather than checked only just before an audit.
DevOps engineers understand that Git remains the expected source of truth.
And the business gets something even more important: fewer infrastructure-related surprises.
Configuration drift is neither spectacular nor particularly exciting when it first appears. It becomes much more interesting three months later.
Especially during an outage.
Infrastructure That Enforces Its Own Rules
Infrastructure as Code provides repeatability.
Policy-as-Code provides enforceable rules.
AI can add context, prioritization, and faster remediation.
Combining these elements creates an interesting model: infrastructure that not only knows how it is supposed to be configured, but continuously compares those assumptions with reality and can respond when discrepancies emerge.
The goal is not to create an environment in which humans are no longer needed.
Quite the opposite.
The goal is to create an environment where engineers do not spend Monday morning discovering that production stopped matching its Terraform configuration sometime on Friday evening.
Problems are best fixed at an early stage — preferably before the first coffee.
Especially the ones listening on 0.0.0.0/0.


