AI and Regression Testing: Modernize your IBM i Applications safely

Generative AI is changing the game on IBM i: analyzing a thirty-year-old RPG program, explaining its logic, suggesting a refactor… All of this can now be done in minutes. It’s a historic opportunity for a legacy estate that often carries the heart of the business: billing, payroll, inventory…

But one question keeps coming up in the boardroom: “Does the modernized application do exactly what it did before?” Not roughly. Exactly. Because on these systems, a difference in behavior means an incorrect invoice or a payroll error.

Our conviction: the safest modernization is hybrid: AI to understand, determinism to transform, regression testing to prove it.

But a language model is probabilistic by design: it produces the most likely output, not a guaranteed one. Ask it to convert the same program twice, and nothing guarantees you’ll get identical results. And across an application portfolio, whether it’s a few hundred programs or several thousand, nothing guarantees that the same conversion rule will be applied consistently everywhere.

This isn’t a flaw in AI. It’s its nature. And that’s precisely why AI alone can’t be responsible for modernizing a mission-critical system.

In these systems, everything comes down to the details: a rounding rule in accounting, a sort order, a numeric truncation context, a legacy date format. Invisible in a code review. Very visible in production.

Faced with this, many organizations conclude: “then everything needs to be reviewed by humans.” That overlooks a third path, proven long before the AI wave: code transformation through deterministic engines, based on syntactic and semantic analysis of the language, not on probabilities.

A deterministic process offers three properties that no generative model can guarantee:

1. Reproducibility: the same input always produces the same output;
2. Consistency: the same rule applies everywhere, without exception;
3. Auditability: every transformation is traceable and can be justified to an auditor.

On IBM i, two major modernization projects can be carried out this way:

Deterministic transformation or not, one governance rule remains: every modernization must be proven. And that proof follows a very precise logic, one that is also deterministic:

That’s the role of ARCAD Verifier. And one point deserves particular attention from QA managers: thanks to the link maintained between application code and test cases, Verifier identifies only the tests affected by a given change. In practice, after converting a batch of programs, you don’t blindly replay 3,000 scenarios: you replay the ones that touch the transformed code. Verification becomes targeted, fast, and therefore genuinely practical at every iteration, not just at the end of the project.

Same input + same scenario = same result. Otherwise, it’s a regression. Trust isn’t declared: it’s demonstrated, screen by screen, record by record, report by report.

Putting the three pieces together, the secure modernization chain looks like this:

AI (understand) → Deterministic engines (transform) → Regression testing (prove) → Trust (deploy)

One question remains: how do the AI agent and the deterministic engines communicate?

That’s the role of the ARCAD MCP Server. Through the standard MCP protocol, it exposes the ARCAD repository (34 years of deterministic analysis of the application estate: components, fields, procedures, dependencies) as more than 70 tools (resources, actions, methodological prompts). The AI agent (Copilot, Claude, an on-premise LLM…) queries these tools in natural language, chains calls together, and can trigger the underlying tooled processes (launching a Free Form conversion through Transformer RPG, for example), all while relying on a certified context derived from deterministic algorithms, not inference.

Three direct consequences for a CIO:

In other words, the hybrid approach described in this article isn’t a concept: it’s an operational architecture, where the AI agent orchestrates and the deterministic engines execute.

This division of labor has an economic consequence that few organizations have measured yet: token consumption.

Having an LLM convert millions of lines of RPG means sending millions of lines in and receiving just as many back out, then starting over with every correction. That means direct cost, latency, energy footprint, and, depending on the architecture, exposure of source code to external services. In the hybrid approach, by contrast, bulk transformation consumes no tokens at all: it’s carried out by syntactic analysis engines. AI is deployed surgically, on high-value tasks: understanding, explaining, documenting, and adjudicating complex cases.

The result: predictable, minimal AI costs, tight control over code confidentiality, and a budget that funds the transformation itself rather than rounds of inference. The MCP protocol amplifies this effect even further: because context makes up for raw model power, even a small local SLM hosted on-premises produces relevant results: sovereignty without the added cost.

Rather than a pitch, here are three situations that CIOs and CISOs will recognize.

Every IBM i estate has its own history, its sensitive areas, its level of technical debt, and its regulatory constraints. There’s no universal modernization path, but there is a method for building your own: assess what you have, choose your projects (Free Form, web services, database), and build the proof loop suited to your delivery cycles.

This is a conversation I have every week with CIOs, CISOs, and application managers. If you recognize your own situation in any of these, or if your leadership team is already asking you for proof, get in touch with me. I’d be glad to discuss your context, share concrete lessons learned, and show you how to bring AI, deterministic transformation, and regression testing together in your own DevSecOps pipeline.