Test Automation and Source Code Analysis for IBM i: why bother enforcing a new quality gate?

By Nick Blamey

What’s in it for the developers and why is it needed for DevOps – a thought inspiring blog by ARCAD’s Director of Northern European operations, Nick Blamey

The business problem:  Continuous Quality of applications depends on implementing policies that enforce immediate validation.  But CIO’s responsible for diverse application assets are lacking code guidelines from where to start measuring code quality and also the resources to allocate on this activity.

Continuous Quality (CQ) and DevOps

Software defects drastically increase the cost of application development. Finding and fixing errors in production is often 100 times more expensive than finding and fixing it during the design and coding phases.  It is vital that teams incorporate quality into all phases of software development and automate quality verification as far as possible to locate defects early in the process and avoid repeat effort.  This is what is meant by “Continuous Quality” or CQ, which forms an essential safeguard – or quality gate – in the rapid delivery cycles in DevOps and CI/CD workflows today.

Which techniques are available for Continuous Quality?

Static code analysis is the simplest and most effective method to prevent defects and harden code while accelerating application delivery.

Automating the code analysis as early as the build or Continuous Integration phase means your team can find and fix systemic defects when the cost of remediation is at its lowest.  After the initial investment in configuring rules and metrics the gains in efficiency become exponential over the development lifecycle.

To achieve continuous quality, organizations can employ a number of strategies.

• Static analysis of source code to identify complexity hotspots, deviations from standards, security loopholes, etc.
• Peer reviews to check code written by one’s equals (peers) to ensure it meets specific criteria.
• Unit testing to execute and scrutinize the individual modules, or units, of an application, for proper operation.
• Regression testing to repeat a set of test scenarios on a new application release to identify any deviations from normal operation.
• Monitoring of the application in production to ensure it is operating correctly after each update and at all times.

To be effective in a DevOps environment each of the above techniques must be both automated and continuous, integrated within the CI/CQ/CD workflow.

How important is Source Code Analysis (SCA) for DevOps?

Source code Analysis: what alternatives?

Faced with the challenge of auditing an existing IBM i code base for quality, CIOs have a limited number of choices:

• Complex peer review process: even supported by collaborative tooling, the manual effort involved in a peer review can be difficult to manage across large teams with a range of expertise.
• External audit of source code by experts: lacking in application knowledge, the learning curve for external code auditors is often steep, making this an expensive option with often unquantifiable benefits.
• Continuous source code analysis using an automated solution designed specifically for an IBM i RPG code base

The cost of defects in application development

A study in software quality from Capers Jones in 2008 came to two very important conclusions:

• Development “wastage” and defect repairs (from deployed code) absorbed almost two-thirds of the US software workforce – leaving only one third for productive work on new projects.
• 50% of software development project budgets are spent fixing poor quality code; fewer than 6% of organizations have clearly defined software management processes in place; and software projects of 100,000 function points in size have a failure rate of 65%.

More recent articles on this topic suggest that for many organisations the statistics remain very much unchanged today.

Since DevOps has now taken over as the key driver in most development shops there is a massive potential for optimisation to eliminate the challenges described above and make developers more efficient in their work by spending more time coding and less time fixing defects.

The limitations of functional testing

Most organizations perform functional testing of their applications through the UI, required for compliance reasons. However, like black box testing, since each and every defect needs to be diagnosed from the UI, this approach brings little information to help developers to actually fix problems. The result is typically a constant stream of defects classified as:

• cannot reproduce the issue
• test environment not set up correctly
• require more information

With poor information for developers, the challenges are pushed downstream. Projects face delays due to lengthy code understanding and a sub-optimal debugging approach. This severely limits the ability for any IBM i development team to maintain the “speed of delivery” required for meeting their DevOps targets.

Static Code Analysis: Who does it and why

There are 3 main approaches to static code analysis in the multi-platform world: Static Analysis for Security, Static analysis for Code Complexity and Static Analysis for Code Quality.

Many products exist to perform this task and the market is large and expanding with a few dominant players. The solutions are often extremely expensive and tend to be less relevant on the IBM i which is less susceptible to security issues than other platforms.

IBM AppScan Source is the best example of the market leader for Code Security but MicroFocus also offer the Fortify Security Suite with a number of additional tools available from other vendors e.g. CheckMarx, Klocwork, CA Veracode.

For Code Complexity metrics, the key players include CAST and McCabe but neither offer support for RPG on the IBM i.

Why do IBM i Developers need Source Code Analysis?

Given the multiple variants of RPG and the sheer longevity of applications, developers on IBM i face a unique challenge with legacy code bases containing millions of lines of code that have been maintained for sometimes thirty years by successive developers. It is laborious to understand program logic and assess the quality of code – resources are diverted to address the “technical debt” of the code base. The challenge is greater still given the ever-growing shortage of RPG skills in the market. The new Free Form RPG syntax has changed the game, offering a means of onboarding a new generation of developers – making the conversion of RPGLE applications to Free Form the “burning platform” of our day.

Source Code Analysis as key part of any “legacy code base Audit”

Source Code Analysis has the potential to be delivered as part of a wider code audit process. Companies like ARCAD have built solutions that generate a complete metadata overview of the entire code base, enabling a deeper level of analysis and integrity checking. Here source code analysis is delivered as part of the code audit and rules and metrics are used to enforce local standards.

ARCAD CodeChecker can create a Code Quality Baseline from which the RPG Code Base can be continually improved through accurate and regular measurement of code quality. This allows CIOs and Development leads to show application owners that they are consistently delivering against ISO 27001 continual improvement goals of the wider organisation.

Widen your net to catch Code Quality issues for RPG

As described above, RPG is a special case in the development world. Among the standard source code analysis tools, a few (such as SonarCube) are able to perform a simple RPG Code Review and Static Quality Analysis, but they are severely limited in their coverage (for example, lacking support for the many RPG variants) and the number of rules they can enforce (limited to around 30 mainly code documentation guidelines).

The potential business risk of these limited tools is that:

• They are not really usable for code quality guideline enforcement for RPG specifically
• They tend to create a number of false positives which limits their effectiveness and could in theory eliminate any value introduced in the Peer Review process by forcing developers to debug issues which are caused by the tool itself.

Modern DevOps organizations are now looking for an “industrial strength” solution to this challenge to ensure that an implementation an open source Source Code Analysis tool doesn’t itself become a bottleneck in the DevOps workflow.

The design goals of ARCAD Code Checker from ARCAD have been therefore to fit with modern large DevOps oriented IBM i RPG development teams, emphasizing:

• Rapid scan of an entire code base
• Auto-Tune the Quality rules for enforcement on a code base by code base, stage by stage basis
• Provide real value to the developers performing the coding work in rapid feedback to the standard of their work after each and every edit. (see section below)
• Seamlessly integrate into a wider ARCAD DevOps tools chain offering, Rapid and complete x-referencing and Auditing, Source Code Management, Dependency Building, Automatic Testing (with deep dive diagnostics of errors) and Deployment and Release Automation for IBM i LPAR environment management.

RAPID VALUE from ARCAD CodeChecker for your CodeBase

Productivity gains through Source Code Analysis Quality Gate enforcement

Typically, if a developer knows within a few minutes of writing code that a guideline has been breached with a desktop code checking product like ARCAD CodeChecker, they can fix issues immediately with minimal impact on quality. If Code is “peer reviewed” a developer could be waiting for days even weeks for feedback by which time the developer has moved to other tasks. The best analogy for this would be a grammar checker with word processing i.e. if you know immediately as you write text that you have made a grammar error, you can fix it immediately whilst the sentence is still in your mind – compared with the time it takes if you perform a grammar check 2 weeks after you wrote the sentence. In this case of course the writer of the text will spend 90% of the grammar check and correction time trying to understand the context of what he/she wrote 2 weeks before.

Driving enforcement of Code Quality by offering real and immediate value to Developers

Many Source Code Analysis tools have a bad reputation. CIOs and Development management must constantly make risk management decisions between: slowing down the development process and the resultant business owner pressure to maintain the speed of delivery vs. introduction of technical debt which could cause a large business risk in the future if Code Quality Guidelines are not enforced.

Code Checker from ARCAD has been designed to add immediate value right at the developer’s workspace / desktop through its integration with RDi and SEU. ARCAD designed Code Checker to cope with the regular comments from RPG developers “If you are going to Mark my homework, at least time me how you are going to judge my success of failure”.

In addition to the optimisation of the Peer Review process through Automatic Code Quality Static Analysis, ARCAD CodeChecker can offer value to developers and your inflight projects:

• Creation and enforcement of Source Code Quality guidelines delivered automatically eliminates the need for Peer Review: allowing developers to focus on Coding more quickly vs peer review.
• Putting in a process through Source Code Analysis for the elimination of technical debt means that IBM i RPG team can stay up to speed with other teams within organisation in a DevOps world.

Combining Source Code Analysis with Testing Automation, Database integrity checking and helping Developers to debug complex issues more rapidly.

Modern Development Teams face this problem

DevOps Bottleneck effect from manual Source Code Analysis, Testing, and Debugging, re-test cycle.

To cope with an acceleration of the DevOps cycles from a few releases per year to more regular releases i.e. monthly or even weekly releases, organizations are driven to perform more regular testing, normally delivered through automation.  They are also impelled to eliminate a lot of the effort which goes into the “localisation of defects” to their root cause, to ensure that developers can drive higher quality code without an impact on the timeframes required by application owners.

Shift left as Key Driver for DevOps:

The graph below shows a typical IBM i / RPG Defect graph i.e. the amount of defects which occur over the time from the start of the project to the actual release date.

Cost of defects across the development lifecycle

Though typically 85% of defects are introduced in the early coding phases of the DevOps cycle, the cost to repair defects grows exponentially through the later phases of test and delivery, reaching inestimably high costs when a defect is found in production, with potentially a significant impact on business bottom line and reputation.

It is clear to see that by “shifting left” the detection of defects, their cost and impact is minimized.

ARCAD Software, through their work with many development teams on RPG have seen that developers perform a number of tasks to advance the detection of errors. These include:

• Hand Coding unit tests themselves to exercise individual program functionality and make sure that they haven’t created defects as they develop.
• Testing individual Batch processes are still working after any changes are made to specific programs.
• Re-setting and working with complex Test Data including Anonymisation requirements.
• X referencing defects across a multitude of components / programs to understand the impact of each code change across other RPG programs and also NON -IBM i x references.
• Scripting the deployment of new code once compiled onto the different LPARS (dev, QA, Prod etc.) and then performing a manual check that once deployed each of the LPARS is fully functioning. This process is typically referred to as “test environment assurance”.
• Preparing the LPAR for a full end to end test execution including load testing and end to end functional testing.

Yet from ARCAD’s experience, each of these processes when performed manually causes additional cost, effort and risk of bottleneck in a DevOps deployment process.

To cope with these challenges, ARCAD offer a number of tools in addition to CodeChecker (Source code analysis) to eliminate bottlenecks in the DevOps process and provide a frictionless process from Functional Specification through to Coding, Unit Testing, Compilation, Build, End to end Functional Test and Production deployment :

• ARCAD Verifier (BATCH and UI Testing)
• ARCAD DOT Anonymizer,
• ARCAD Observer for X-referencing and
• ARCAD Builder and Drops for deployment automation and Release Management.

Each of these solutions can add additional value to your Development Process, shifting left to reduce overall cost in the development cycle:

Contribution of ARCAD solutions to a “shift left” of development costs

ARCAD view and positioning

As a company ARCAD began its evolution fixing the Source Code analysis problem of Year 2000 date format changes. Since then ARCAD have provided solutions to the most burning and current challenges our 350+ customers face with their RPG code bases: X referencing, Auditing, Source Code Management, Building, Testing and Deploying.

ARCAD for DevOps:  suite of solutions integrated over a repository core

Suggested next step:

An Audit process using ARCAD expertise and tooling is an excellent starting point to your journey to a quality DevOps process.

To find out more about how ARCAD have designed their solutions to fix the next problem in Source Code Analysis. Contact ARCAD and see how CodeChecker and other ARCAD DevOps tools for IBM i can help with your CodeReview, Audit, Testing and DevOps processes.

Further reading:

• “the business value of Source Code Analysis”: sponsored by IBM.
• Brian Chess
• Shift Left Testing definition