By Jeff Tickner <\/h3>\n<\/div><\/div><\/div>Every IT organization does some kind of testing – whether it is unit testing by developers, QA testing by QA staff or full-on automated regression testing. In each of these techniques, one area that is largely underestimated is the role of test data<\/strong>. Yet how effectively this data is managed has a direct impact on the time spent in testing and the quality of the results. We will look at why test data can be hard to manage, what are the best practices based on our experience and which tangible benefits you stand to gain from a consistent set of test data.<\/p>\n<\/div>Summary<\/h2>\n\n- Why does test data matter? <\/a><\/li>\n
- Test\u00a0data matters, even for unit tests!<\/a><\/li>\n
- Consistent data is the foundation of functional\u00a0regression testing<\/a><\/li>\n
- How to ensure\u00a0the\u00a0privacy\u00a0of test data\u00a0\u2013 and refresh it when needed?<\/a><\/li>\n
- To sum up:\u00a0 continuous test (CT) needs consistent data!<\/a><\/li>\n<\/ol>\n<\/div><\/div>\n
1. Why does test data matter?\u00a0\u00a0<\/span><\/h2>\nLet's\u00a0talk\u00a0first\u00a0about the foundation of testing, the test data set. Everyone understands that having data to test against is a requirement, but how should that data be managed? Too often I see staff testing against data that\u00a0becomes\u00a0more and more corrupted as testing progresses\u00a0causing\u00a0friction in the testing process. When a test fails or does not produce the expected results,\u00a0users have to determine\u00a0whether\u00a0those\u00a0erroneous results were caused by the program changes they are testing\u00a0or\u00a0instead\u00a0by\u00a0data that is bad or shifting\u00a0while\u00a0they are testing. Most commonly they report back to a developer\u00a0pointing out\u00a0a defect, who\u00a0then has\u00a0the challenge of trying\u00a0to reproduce\u00a0the error\u00a0\u2013\u00a0impossible\u00a0if\u00a0they are\u00a0using a\u00a0'better'\u00a0set of data. You can\u00a0easily\u00a0see how bad data can have a snowball effect on wasting time in testing.\u00a0So how can we manage data to keep testing efficient?<\/p>\n
Before we\u00a0dig deeper into\u00a0data\u00a0let's look closer at\u00a0testing itself.\u00a0We should distinguish between\u00a0two main types,\u00a0unit testing and functional testing\u00a0(commonly used for regression testing).\u00a0Arcad\u00a0has experience with both\u00a0types\u00a0(since\u00a0we\u00a0have tools for both)\u00a0but we aren't here to talk about our tools but one\u00a0of the\u00a0most\u00a0critical aspects of\u00a0good testing:\u00a0good data<\/strong>.\u00a0Our tools have the same dependency as any other testing, bad data = bad testing.<\/p>\n<\/div>\n2. Test\u00a0data matters, even for unit tests!<\/h2>\n
Unit testing is normally very granular and involves calling a program or procedure with a set of parameter values and checking the return parameters for the correct values. However,\u00a0some programs may chain out to files to validate or read additional data before returning values.\u00a0So\u00a0while unit testing is not normally considered data dependent,\u00a0in certain cases\u00a0(and especially\u00a0with legacy applications developed on\u00a0the IBM\u00a0i\u00a0platform)\u00a0we come across\u00a0monolithic programs that have\u00a0many\u00a0business functions behind the input and output\u00a0parameters.\u00a0If you don't have consistent data for your programs to consume during unit testing you may not get consistent results. Unit testing tools often have a provision for\u00a0the data that will be consumed by the program can be updated before the unit test is executed.\u00a0\u00a0Or if\u00a0not, they\u00a0can be scripted\u00a0to\u00a0cater for this data.\u00a0\u00a0However,\u00a0getting this provisioning right\u00a0is\u00a0not only\u00a0time consuming\u00a0but may\u00a0require technical skills\u00a0in scripting\u00a0a file update for the unit test.<\/p>\n<\/div><\/div><\/div>
Your unit testing is tedious or incomplete? <\/strong>Automate with ARCAD iUnit !<\/strong><\/p>\n<\/div>Download the datasheet<\/span><\/i><\/a><\/div><\/div><\/div><\/div><\/div>3. Consistent data is the foundation of functional regression testing<\/h2>\n
Functional testing is based on executing user-level business functions, just like the test cases that are developed to test and verify that a specific function is operating correctly. Often functional tests are scripted or executed and recorded in a tool (like Arcad Verifier<\/a>) and the results are checked for consistent values. Once these functional tests are developed they can be used again and again for regression testing, in which one or more functional tests are executed to validate normal application operation after an update to QA. This technique is very dependent on having consistent data so that the functional tests return consistent results and pass.<\/p>\nIf the data is changing due to other processes running while the scripted tests are being executed, a false positive can occur that will be interpreted as failure. So here again consistent data = consistent results and enables a much higher level of automation. The importance of having a dedicated set of test data becomes even more apparent.<\/p>\n<\/div><\/div><\/div>
Data-oriented Test Automation for IBM i. ARCAD Verifier isolates defects as they arise, deep in your data – for interactive, batch & Webservices.<\/strong><\/p>\n<\/div>Download the datasheet<\/span><\/i><\/a><\/div><\/div><\/div><\/div>\n4. How to ensure the privacy of test data \u2013 and refresh it when needed?<\/h2>\n
The most common reason why businesses do not have multiple sets of test data is because the starting point is production data which can have millions of records and\/or sensitive data. Arcad has a tool to handle that case as well (DOT Anonymizer<\/a>) but that is another topic and we are here to talk about managing that data to your advantage. If you build a good set of test data that is big enough to support all of your test cases but no bigger, it may contain sensitive data that is of high value to your organization and which requires masking and otherwise careful handling. Having a copy of that set aside for refreshing QA environments ideally in a SAVF so it cannot be inadvertently updated is a recommended best practice. Alternatively building a function to refresh that data from production (or using an Arcad tool) is an option and will also provide a means of refreshing test data with more recent production data. Either method means that when a set of test data gets corrupted it is easy to refresh it with good data and, in the case of automation, with consistent values.<\/p>\nThis ability to automatically refresh data is valuable for all the testing you do, even developer-driven unit testing which may be dependent on data read from files. Certainly, functional testing by QA needs the ability to refresh their data easily since they are trying to uncover bugs that might corrupt their test data as they are testing. Even using testing tools that manage data updates through a rollback process can leave behind bad data if the QA staff hits a significant error and the testing script process is interrupted.<\/p>\n
Every IT organization does some kind of testing – whether it is unit testing by developers, QA testing by QA staff or full-on automated regression testing. In each of these techniques, one area that is largely underestimated is the role of test data<\/strong>. Yet how effectively this data is managed has a direct impact on the time spent in testing and the quality of the results. We will look at why test data can be hard to manage, what are the best practices based on our experience and which tangible benefits you stand to gain from a consistent set of test data.<\/p>\n<\/div> Let's\u00a0talk\u00a0first\u00a0about the foundation of testing, the test data set. Everyone understands that having data to test against is a requirement, but how should that data be managed? Too often I see staff testing against data that\u00a0becomes\u00a0more and more corrupted as testing progresses\u00a0causing\u00a0friction in the testing process. When a test fails or does not produce the expected results,\u00a0users have to determine\u00a0whether\u00a0those\u00a0erroneous results were caused by the program changes they are testing\u00a0or\u00a0instead\u00a0by\u00a0data that is bad or shifting\u00a0while\u00a0they are testing. Most commonly they report back to a developer\u00a0pointing out\u00a0a defect, who\u00a0then has\u00a0the challenge of trying\u00a0to reproduce\u00a0the error\u00a0\u2013\u00a0impossible\u00a0if\u00a0they are\u00a0using a\u00a0'better'\u00a0set of data. You can\u00a0easily\u00a0see how bad data can have a snowball effect on wasting time in testing.\u00a0So how can we manage data to keep testing efficient?<\/p>\n Before we\u00a0dig deeper into\u00a0data\u00a0let's look closer at\u00a0testing itself.\u00a0We should distinguish between\u00a0two main types,\u00a0unit testing and functional testing\u00a0(commonly used for regression testing).\u00a0Arcad\u00a0has experience with both\u00a0types\u00a0(since\u00a0we\u00a0have tools for both)\u00a0but we aren't here to talk about our tools but one\u00a0of the\u00a0most\u00a0critical aspects of\u00a0good testing:\u00a0good data<\/strong>.\u00a0Our tools have the same dependency as any other testing, bad data = bad testing.<\/p>\n Unit testing is normally very granular and involves calling a program or procedure with a set of parameter values and checking the return parameters for the correct values. However,\u00a0some programs may chain out to files to validate or read additional data before returning values.\u00a0So\u00a0while unit testing is not normally considered data dependent,\u00a0in certain cases\u00a0(and especially\u00a0with legacy applications developed on\u00a0the IBM\u00a0i\u00a0platform)\u00a0we come across\u00a0monolithic programs that have\u00a0many\u00a0business functions behind the input and output\u00a0parameters.\u00a0If you don't have consistent data for your programs to consume during unit testing you may not get consistent results. Unit testing tools often have a provision for\u00a0the data that will be consumed by the program can be updated before the unit test is executed.\u00a0\u00a0Or if\u00a0not, they\u00a0can be scripted\u00a0to\u00a0cater for this data.\u00a0\u00a0However,\u00a0getting this provisioning right\u00a0is\u00a0not only\u00a0time consuming\u00a0but may\u00a0require technical skills\u00a0in scripting\u00a0a file update for the unit test.<\/p>\n<\/div><\/div><\/div> Your unit testing is tedious or incomplete? <\/strong>Automate with ARCAD iUnit !<\/strong><\/p>\n<\/div> Functional testing is based on executing user-level business functions, just like the test cases that are developed to test and verify that a specific function is operating correctly. Often functional tests are scripted or executed and recorded in a tool (like Arcad Verifier<\/a>) and the results are checked for consistent values. Once these functional tests are developed they can be used again and again for regression testing, in which one or more functional tests are executed to validate normal application operation after an update to QA. This technique is very dependent on having consistent data so that the functional tests return consistent results and pass.<\/p>\n If the data is changing due to other processes running while the scripted tests are being executed, a false positive can occur that will be interpreted as failure. So here again consistent data = consistent results and enables a much higher level of automation. The importance of having a dedicated set of test data becomes even more apparent.<\/p>\n<\/div><\/div><\/div> Data-oriented Test Automation for IBM i. ARCAD Verifier isolates defects as they arise, deep in your data – for interactive, batch & Webservices.<\/strong><\/p>\n<\/div> The most common reason why businesses do not have multiple sets of test data is because the starting point is production data which can have millions of records and\/or sensitive data. Arcad has a tool to handle that case as well (DOT Anonymizer<\/a>) but that is another topic and we are here to talk about managing that data to your advantage. If you build a good set of test data that is big enough to support all of your test cases but no bigger, it may contain sensitive data that is of high value to your organization and which requires masking and otherwise careful handling. Having a copy of that set aside for refreshing QA environments ideally in a SAVF so it cannot be inadvertently updated is a recommended best practice. Alternatively building a function to refresh that data from production (or using an Arcad tool) is an option and will also provide a means of refreshing test data with more recent production data. Either method means that when a set of test data gets corrupted it is easy to refresh it with good data and, in the case of automation, with consistent values.<\/p>\n This ability to automatically refresh data is valuable for all the testing you do, even developer-driven unit testing which may be dependent on data read from files. Certainly, functional testing by QA needs the ability to refresh their data easily since they are trying to uncover bugs that might corrupt their test data as they are testing. Even using testing tools that manage data updates through a rollback process can leave behind bad data if the QA staff hits a significant error and the testing script process is interrupted.<\/p>\nSummary<\/h2>\n
\n
1. Why does test data matter?\u00a0\u00a0<\/span><\/h2>\n
2. Test\u00a0data matters, even for unit tests!<\/h2>\n
3. Consistent data is the foundation of functional regression testing<\/h2>\n
4. How to ensure the privacy of test data \u2013 and refresh it when needed?<\/h2>\n
![\"Philippe](\"https:\/\/www.arcadsoftware.com\/arcad\/wp-content\/uploads\/2020\/05\/jeff-tickner.jpg\" "\\"Jeff")
<\/span><\/div><\/div><\/div>