How Test Automation combined with Batch Testing and Defect Analysis at the Database level can optimize your DevOps process.<\/em><\/strong><\/p>\n With full end-to-end testing through the User Interface<\/strong> already established at many organisations, the challenge of \u201cshifting-defects-left<\/strong>\u201d in legacy applications still exists. Service Virtualisation is an option to test multi-tier applications with a legacy<\/strong> core but this level of abstraction can often leave a substantial body of mission critical application logic unchecked, presenting a significant risk to your project. This article explores a several options available to resolve these challenges.<\/p>\n A thought- inspiring blog by ARCAD\u2019s Director of Northern European operations, Nick Blamey<\/em><\/p>\n<\/div> Time-to-market and the associated benefits of providing rapid feedback to developers are a key part of any DevOps strategy<\/strong>. Standard approaches of catching defects which occur at the UI layer<\/strong> \u2013 while important \u2013 still fail to eliminate a number of key challenges organisations face in particular in the following situations:<\/p>\n Most organisations are embracing End to End UI Testing automation<\/strong> using a combination of the following solutions:<\/p>\n Each Test Automation<\/strong> solution has advantages and unique capabilities to support different Testing Approaches but they all lack the ability to test, locate (triage) and fix complex legacy application errors buried deep in the legacy<\/strong> applications themselves on the IBM i (AS\/400).<\/p>\n We can analyse the 3 main challenges<\/strong> and potential bottlenecks<\/strong> to the process which cannot be eliminated by these standard tools as follows:<\/p>\n From ARCAD\u2019s experience with some of the largest Legacy application estates, these challenges are becoming more and more complex to solve. The rest of this document will explain both why these challenges have a potential to limit the speed of your DevOps process but also how you can solve these challenges using ARCAD solutions.<\/p>\n The goal of the remainder of this article is to explain in detail both the options at your organisation\u2019s disposal to resolve these problems, and the solutions available from ARCAD proven by multiple software development teams, typically developing on a combination of IBM i legacy applications and the front-end apps leveraging the business logic contained within the legacy.<\/p>\n The most common defects created by Batch programs tend to be created by simple mistakes in the configuration of Spool and Database commands. However simple it is to \u201cfall foul\u201d of these simple but potentially \u201cshowstopping\u201d defects, the effort most teams tend to expend in trying to reproduce them and fix them can massively limit the ability for a development team to make risk-free changes to applications where Batch, Spool and database edits form a crucial part of their functionality.<\/p>\n Typical Testing Challenges with Spool, Database and Batch can be classified as follows:<\/p>\n These challenges above can all be handled seamlessly and most importantly with zero<\/strong> programming effort using the ARCAD-Verifier Test Automation solution, described below.<\/p>\n To ensure that a \u201ctestable\u201d application is passed to the QA team from development in rapid iterations many organisations create a suite of Build Verification Tests (BVT). This process is key to ensuring that defect tracking systems are not filled with \u201ctest environment unavailable\u201d or \u201cincorrect data in Test Environment\u201d defects.<\/p>\n Smoke testing covers most of the major functions of the software but none of them in depth. The result of this test is used to decide whether to proceed with further testing. If the smoke test passes, go ahead with further testing. If it fails, halt further tests and ask for a new build with the required fixes. If an application is badly broken, detailed testing might be a waste of time and effort.<\/p>\n The risk to any organisation of NOT having a BVT \/ Smoke Testing process is a large number of additional defects being created, each of which need to be analysed, triaged and fixed even when the defects are actually NOT software problems but rather Build Verification problems.<\/p>\n The effect of lack of Build Verification Testing is that a typical defect classification Report Tends to show many defects classified as \u201ccannot reproduce error\u201d or \u201ctest environment issue\u201d or \u201ctest data issue\u201d thereby slowing down development time but also distorting the ability for Software development organisations and application stakeholders to make rational decisions about the severity of particular defects.<\/p>\n The biggest challenges facing IBM i developers is that a \u201cbatch\u201d workload by definition interacts with and impacts many components \/ programs and is often critical functionality that runs \u201cheadless\u201d i.e. in an unconstrained mode.<\/p>\n A batch job will always complete its workload in the shortest possible \u201clow application usage\u201d window to eliminate the impact on application performance. The challenge this creates for IBM i developers is that a batch process attempts to override any other workload running simultaneously because it will always prioritize execution speed over any resource balancing on the hardware it is running on. The only time a Batch process will lose priority is when its execution is limited by a system bottleneck i.e. normally the smallest Data Flow point in the application.<\/p>\n Because of the two problems detailed above, many developers look to perform scheduled batch jobs out of hours which can limit the functionality of the application whilst users wait for the \u201covernight batch jobs\u201d to complete before performing their work.<\/p>\n Modern DevOps development teams on the IBM i have to constantly manage the following challenge: the need for batch jobs to execute quickly due to their \u201cmission criticality\u201d of their work and the reliance of other application functionality on their completion.<\/p>\n Whilst many Batch applications on the legacy were written many years ago, by definition they often contain mission critical functionality constituting a major part of your application\u2019s value. For these reasons the continual enhancement of batch functionality presents a number of challenges:<\/p>\n For this reason, many organisations attempt to create a totally seamless process for Batch Process change\/commit and then attempt to make these changes available to the Test LPAR.\u00a0 The effort involved in trying to perform Batch testing can then become punitive with the maintenance of the batch test process over-consuming development team resources and causing project delays.<\/p>\n Because of the complexity of testing Batch and their specific data and test environment requirements, batch failures have led to some of the most publicized incidents ever witnessed. A particularly high-profile incident was the prolonged downtime at RBS in 2015 – and a similar episode in 2012 – caused by batch process failures at the bank impacting hundreds of thousands of customers.<\/p>\nSummary<\/h2>\n
\n
The business problem<\/h2>\n
\n
\n
A different paradigm for testing and locating defects with Legacy Systems<\/h2>\n
\n
Why are Batch Testing, Database Level Testing and Defect Triage each a major potential challenge and potential bottleneck to your development process?<\/h3>\n
\n
“Unpicking the Spaghetti of Batch processes”: Common Defects from Batch are easily diagnosable in the Spools and Database<\/h3>\n
\n
DevOps requires Build Verification Testing (BVT) \/ Smoke Testing<\/h3>\n
Problems Associated with Batch Processing<\/h2>\n
Batch testing: make sure that your batch performance does not cause defects downstream by non-completion<\/h3>\n
\n
Why direct your testing efforts onto Batch and Database level testing?<\/h3>\n
\n
How severe can Batch defects become?<\/h3>\n