5. Integrate your delivery mechanisms
We already had an automated process for integrating changes with test environments but it was unbelievable to me that we didn’t have something automated for production. With the ARCAD DROPS tool we did have something there already which could be used for this, it was just an extension of our existing configuration. I figured we would just give it a try one Sunday and see how it works out.
I don’t normally spend my Sunday mornings working on putting changes into the production application, but this Sunday was different. We were all there, the development Pod manager, me, and our most experienced change management guru to actually deliver the changes via a SAVF (Save File). We were warned that it could be the most boring 3 hours of our life as that’s how long it was expected for this volume of changes to be delivered to production.
My coffee hadn’t even gone cold and…“I think we’re finished”, we were 30 minutes into what was supposed to be a 3 hour marathon delivery. “Well that went well”…I said as DROPS had brought out the master of understatement in me.
The delivery was fully documented, we had lists of objects that had been changed, we had details of the linked programs and files that had been rebuilt. Traceability of the changes was incredible and we could tie it all back to the original work ticket through the ARCAD hooks into our Jira ticketing system. We even had a sophisticated roll-back feature.
6. Build a central source code repository (with Git)
We installed the full source code for all relevant applications on each Pod members laptop (RPG is very compact so no space issues). The development Pod was now able to spend more time working from home which gave them more flexibility to manage their time and add balance back into their lives, not to mention a morale boost. We still did the daily scrums virtually to make sure everyone was up to speed with the progress of the sprint, but the need to be in the same office had been dramatically reduced.
Moving to a central source code repository and having the Pod members commit their changes rather than needing to be constantly online to the IBM i virtually eliminated telecommunication grief. This grief would happen daily when the neighborhood kids all returned home from school at the same time and overloaded the telecommunications bandwidth. Also when workmen periodically shut off power for a couple of hours forcing you to contemplate driving to the local coffee house just to get back online.
We implemented Git (the industry standard distributed version control system for tracking changes) and it worked really well here. The interface with ARCAD Builder and some additional ARCAD plugins made linking the source code repository and applications on the IBM i really easy.
YouTube was a great resource for helping the Pod to understand Git basics. Learning about branching and merging from a large global community of coders was liberating for the Pod. Adopting Git and ARCAD for our RPG coding made more sense to everyone now. We were starting to leverage the collective knowledge of a much larger community. We were starting to share code with other IBM i developers elsewhere in our global company. We were no longer stuck in a stuffy room hiding behind a bimodal shield of resistance. Since we were now starting to use modern tools and methods, it enabled the Pod to feel more secure about their jobs and their contribution to the company and it allowed us to appeal to new staff.
7. Automate your pipeline
At a certain point in this journey I noticed that the benefits were starting to roll in faster than our efforts rolled out. We were reaching that point where some benefits were starting to roll in for free as welcome byproducts of what we had already done.
The Pod, now in frequent contact with a global community of coders, had started to run with DevOps themselves and were coming back to me with more ways to improve the development cycle and remove more blockers.
ARCAD had customizable build and integration macros chained together, hence after a successful build we were delivering application changes automatically into Test environments with the ARCAD macros. It was a nice solution; it was a good stepping stone, but I now had a Pod hungry for full-DevOps so I knew that even more DevOps automation would be needed soon.
ARCAD has a reliable integration with Jenkins, so after some discussions with the Pod we returned the ARCAD macros to their original (non-customized) state so that they would remain fully compatible with Jenkins moving forward. We then implemented the natural Jenkins pipeline to integrate and deliver our builds into our test environments. A Jenkins CI (continuous integration server) was now running the development pipeline for our heritage AS/400 development, bang! This was a goal from the start and when it went live it gave the whole Pod a real sense of achievement. We now had a paved “highway” for integration.
8. Bring automated testing into your pipeline
We now realized that we were delivering changes to test environments so fast it didn’t allow enough time for even basic unit testing. This resulted in a lot of errors returned to the Pod and this in turn took time out of the sprints. We had inadvertently introduced a bottleneck exactly where we didn’t need one. We were spending too long testing the changes which and this was causing problems when delivering improvements to customers (yes. I eventually convinced the Pod to say customers not users:)) against an agreed timeline.
There is nothing else out there that comes close to off-the-shelf ARCAD capabilities when it comes to unit test automation unless you put in a whole lot of effort to rebuild a decade old piece of open source. ARCAD iUnit plugged straight into our existing pipeline. The Pod was writing their own ARCAD iUnit test cases and sharing them within a day or two. The test cases tied back to specific components so that we could unit test just the specific functions that had changed rather than having to unit test an entire program (we have large programs).
The Pod started using ARCAD Verifier to automate regression testing and they concentrated on building test scenarios which could be reused across various application functions. This reusability was a big time saver. This additional fully automated testing improved application reliability so much that it would have taken 3 times the staff to achieve the same using the previous methods.
The Pod then implemented ARCAD Code Checker plugins and ‘wow’ it’s an incredible tool! It’s not a testing tool, it works on improving the quality of source code in the first place. It enabled us to define and apply coding standards across all source code and it demonstrated where quality weaknesses existed and why. It was like an automated peer review. We were able to enforce code quality standards across the full development Pod. We could for example automatically identify code that would have opened up security issues (extra big deal for a bank) and bounce it straight back to the developer. As time went on the Pod could see code quality issues in near real time and fix it at the root while coding.
By this stage continuous integration and continuous testing had already delivered quantum improvements. Application changes were being delivered more rapidly, more reliably and with better business value for our customers. Customers were commenting that they were seeing their requested enhancements quicker and they were getting improved business benefit from them. This was all as a direct consequence of the DevOps cycle we had implemented.
The value streams we were building with these tools were looking good and there were still things we needed to refine but our “super-highway” was in place.
9. Fast forward 3 years
Yes it took us 3 years but we have achieved a fully automated CI/CT/CD pipeline . We are now continuously delivering IBM i changes to production via an Agile development community and without manual steps. Sundays have been reclaimed and we are able to deliver changes to the business without introducing downtime and this in turn has increased the hours the business can operate, a competitive advantage.
The old monthly release timetable has been replaced with a Value Stream dashboard viewable by managers across our global company. The improvements in application quality is now viewable since various ARCAD tools report up to a graphical quality dashboard.
Our automated testing has resulted in a ‘shift left’ for error catching and fixing and this has made a real improvement to the costs involved in every change and the speed with which they are delivered. All regression testing is fully automatic and unit testing has become a mouse-click.
These robust, continuous micro deliveries are having a positive impact on the business. There are no more monthly change review meetings. Instead, there is more involvement with the business, discussing their pain points, finding out about how we can next evolve the application to create more business benefit for them.
Last but not least all legacy RPG code has been automatically refactored to modern free form RPG using ARCAD Transformer. This was an investment for the future, we have highly experienced expert coders planning their retirements working alongside the brightest and youngest coders using a shared source code repository hosted in a cloud. All of them are developing the applications for our heritage IBM i and following an Agile methodology with an automated pipeline and all under the watchful eye of DevSecOps.
That’s a far cry from the ‘all green screen’ shop 3 years ago..