Most of us know this from personal experience, or have heard similar stories: someone in the IT department makes a decision on Friday to carry out critical changes right before the weekend. Unfortunately, that's not a good day for such actions. Even an innocuous deploy is asking for trouble. And as often happens in life, that was exactly the case at a company providing SaaS-based services, where a new version of the payment module was about to be pushed to production by the DevOps team. A much-needed change, but not on Friday. The change itself wasn't large, but unfortunately it touched a critical business path – payments. And despite assurances that the deployment should succeed successfully, the choice of weekday and the significance of the change already constituted a serious challenge. What if something goes wrong?

The company already used a well-refined CI/CD pipeline, had monitoring, logs, metrics, dashboards, and graphs that would give any manager a sense of security. The problem was that most of these tools determined the actual state only after deployment. If performance decreased, or the error rate went up, the team learned about it only after the change entered production. Sometimes faster, sometimes slower, but almost always – painfully. And that's exactly why Deployment Risk Prediction with Harness Continuous Verification was introduced – a mechanism that verifies deployment based on observability data and detects deviations from the application's normal behavior.

Harness works in a very practical way. After adding a "Verify" step to the pipeline, it can retrieve data from tools like Prometheus, Datadog, Splunk, or AppDynamics. It analyzes them during the deployment process. Then, Machine Learning builds a baseline of the service's normal behavior and compares it with the freshly deployed version. If anomalies occur, the system marks that deployment as risky. There's also a scenario, with appropriate configuration, where the tool can automatically rollback the change. This is Deployment Risk Prediction in practice – predicting risk based on the application's real behavior.

In this specific situation, the payment module was deployed using a <canary> strategy. This is important because Harness Continuous Verification works particularly well when it can compare new instances with those still running the older version. In the first stage, the rollout covered a small portion of traffic. The system collected response metrics, error counts, and log data, comparing them with the baseline. After a few minutes, it turned out that the new version wasn causing an obvious outage. However, it did lead to increased response time and raised the frequency of one type of application error. This "error" was so specific that a human could notice it only after about fifteen minutes of manual chart analysis, assuming the ability to take action in this particular situation.

Harness demonstrated considerable effectiveness and speed in data analysis. ML-based verification detected an anomaly compared to normal behavior, and further deployment progression was automatically halted at the pipeline phase. The team didn't need to perform an emergency rollback after full rollout. There was also no need to wait for customer feedback in the form of support tickets. The problem was stopped, and its potentially negative impact was minimized to the absolute minimum. A seemingly small thing, yet such great relief.

An example pipeline fragment looked like this:

- step:

type: Verify

name: Verify Payment Deployment

spec:

verificationType: Canary

sensitivity: High

duration: 10m

failOnNoAnalysis: true

The whole power lies in simplicity. A few lines of configuration harnessing a mechanism that analyzes deployment data, compares it with historical data, and helps quickly make decisions.

The greatest measurable benefit is time savings. Previously, similar deployments after pushing the change required manual supervision by two people for ~60–90 minutes. One monitored and analyzed in APM, the other checked logs and alerts. After implementing Harness Continuous Verification, the team's active participation stage was shortened to about 15–20 minutes, since most initial analysis was performed by the system. The time savings achieved in this way is ~70–80% for a single deployment of a critical service. Assuming several rollouts per week, this becomes a quite real number of recovered hours for the team.

Regarding financial benefits, that's a completely separate but equally interesting topic, as it doesn't stem directly from work time alone. It's about much more. That slight deterioration in payment module performance could have hit conversion and revenue. To illustrate potential losses, even a 1–2% drop in conversion at a large store translates to tens of thousands of euros less in monthly revenue. Harness describes Continuous Verification as a layer supporting rapid rollback initiation when error rate increases or metrics deteriorate. This significantly reduces the risk of costly changes during each deployment, minimizing manual work for SREs. In the scenario described above, the company avoided a full-scale deployment of a version with an error, and consequently avoided potential losses resulting from transactional problems, team overtime, and later analysis of "what actually went wrong." These non-obvious savings at first glance, once passed through the finance department, are quickly appreciated.

There's also a benefit in terms of reputation. The IT department gained a reputation as a risk-control department capable of detecting problems before customers do. For business and, most importantly, for customers, this is an enormous difference, since quickly detected irregularities in the internal environment represent success in maintaining service continuity, a reputational success, and an organizational success.

But that's not all. Importantly, IT department morale also increased, more than anyone expected. With the changes, nightly emergency rollbacks disappeared, the number of nervous situations after deploy decreased, and there are fewer overworked weekends resulting from incidents – "the team breathed a sigh of relief." Harness itself indicates that every deployment manually verified by SRE doesn't scale well. Continuous Verification systematically transfers that knowledge into the pipeline area. In practice, this means less monotonous checking, fewer errors resulting from routine, less stress, and more sense that the process truly supports people.

The most interesting thing in this whole story is that Deployment Risk Prediction doesn't mean a futuristic AI laboratory. It's simply about a well-used tool that collects metrics, learns the service's normal behavior, and can quickly predict an approaching problem. A bit like an experienced administrator, but without exhaustion and hectoliters of coffee consumed.

Therefore, Deployment Risk Prediction with AI in DevOps is profitable on three levels. In terms of time, because it reduces manual analysis and response time. Financially, because it limits the cost of failures, rollbacks, and deployments with errors. Reputationally, because it builds an image of a team that controls risk instead of having to explain themselves "post factum" every time. Additionally, IT department morale increases, the number of stressful situations decreases, and ultimately this creates not only good technical practice but simply a sensible way of working.