Biblio

In this paper, we present our test automation work applied on national e-health portal for residents in Norway which has over million monthly visits. The focus of the work is threefold: delegating automation tasks and increasing reusability of test artifacts; metrics for estimating efficiency when creating test artifacts and designing robust automated test cases. Delegating (part of) test automation tasks from technical specialist (e.g. programmer - expensive resource) to non-technical specialist (e.g. domain expert, functional tester) is carried out by transforming low level test artifacts into high level test artifacts. Such transformations not only reduce dependency on specialists with coding skills but also enables involving more stakeholders with domain knowledge into test automation. Furthermore, we propose simple metrics which are useful for estimating efficiency during such transformations. Examples of the new metrics are implementation creation efficiency and test creation efficiency. We describe how we design automated test cases in order to reduce the number of false positives and minimize code duplication in the presence of test data challenge (i.e. using same test data both for manual and automated testing). We have been using our test automation solution for over three years. We successfully applied test automation on 2 out of 6 Scrum teams in Helsenorge. In total there are over 120 automated test cases with over 600 iterations (as of today).

We present a process driven test automation solution which enables delegating (part of) automation tasks from test automation engineer (expensive resource) to test analyst (non-developer, less expensive). In our approach, a test automation engineer implements test steps (or actions) which are executed automatically. Such automated test steps represent user actions in the system under test and specified by a natural language which is understandable by a non-technical person. Then, a test analyst with a domain knowledge organizes automated steps combined with test input to create an automated test case. It should be emphasized that the test analyst does not need to possess programming skills to create, modify or execute automated test cases. We refine benchmark test automation architecture to be better suitable for an effective separation and sharing of responsibilities between the test automation engineer (with coding skills) and test analyst (with a domain knowledge). In addition, we propose a metric to empirically estimate cooperation between test automation engineer and test analyst's works. The proposed automation solution has been defined based on our experience in the development and maintenance of Helsenorg, the national electronic health services in Norway which has had over one million of visits per month past year, and we still use it to automate the execution of regression tests.