The following example describes a recommended approach to the implementation of automated software testing, as seen by the author, Bruno Bosshard.
I have most experience with the Java stack, and therefore the following implementation example applies to this technology, but it can also easily be adjusted to Microsoft .NET by replacing the tools appropriately.
I love Java, because it has a large community base and because it is platform independent (supporting Microsoft Windows, Linux, and Apple MacOS).
In Java, the developers will most likely work with an Integrated Development Environment (IDE), such as Eclipse, NetBeans, or IntelliJ IDEA (or Android Studio for Android apps), and that should therefore be the starting point both for Unit testing by the developers and specialised test automation developers that work closely with the developers, as well as all subsequent higher level testing, such as System (Integration) Testing and above.
I would use JUnit (TestNG to be precise) as unit test framework for creating data driven tests. The data for the data driven tests could come from any number of sources, including text files (CSV, XML, JSON etc.), Microsoft Excel, database connections (through JDBC), and many more, as described in the Java Framework document.
For JavaScript applications, I would use Jasmine as (unit) test framework and Karma as test runner to execute the JavaScript tests in Continuous Integration.
As an alternative to Selenium, I would recommend Playwright. It supports interactive web apps by piercing through shadow DOM. It also directly supports web frameworks like React and Vue, and it includes WebKit (Apple Safari) testing on Microsoft Windows, Linux, and MacOS.
I would also strongly consider Cypress as an alternative testing tool to Selenium (and Selenium-based JavaScript frameworks like webdriver.io, nightwatch, or protractor). Cypress has been built from the ground up around JavaScript and Node.js and offers fast, easy and reliable testing for anything that runs in a browser.
Behaviour Driven Development (BDD) extends traditional unit testing and integration testing with sensible automated acceptance testing. BDD formalises and builds on the principles of Test-Driven Development (TDD). TDD reverses the usual sequence of coding first and testing afterwards. In TDD, tests are written in advance of the code needed to satisfy them.
BDD facilitates clearer and unambiguous communication and continuous feedback between the business stakeholders and development and testing. Traditional unit testing is still the foundation of automated testing and is required for fast code checks. The focus of traditional unit testing is on the program code internals (such as statement, branch, and path coverage), while BDD ensures software quality from a business oriented perspective. Traditional unit testing will still have the highest number of test cases and coverage.
I would use Cucumber (or Specflow in .NET) and Gherkin for automated acceptance testing. What makes Cucumber stand out from the crowd of other tools is that it has been designed specifically to ensure that automated acceptance tests can easily be read — and written — by anyone on the team. This reveals the true value of acceptance tests as a communication and collaboration tool. The easy readability of Cucumber acceptance tests through its Gherkin syntax and structure that is optimised for human readability draws business stakeholders into the process, helping developers and testers explore and understand their requirements. BDD ensures that the software design meets the need of the actual code and leaves behind a suite of tests and up-to-date documentation to help preserve the integrity of the code.
The unit tests and integration tests, as well as the automated acceptance tests using BDD (which are still technically unit tests based on JUnit) would be integrated into Apache Maven as Java build management tool. Maven would be used to define project structures, dependencies, builds, and test management.
I would use Jenkins as Continuous Integration (CI) server with a complete scripted Multibranch pipeline that covers program code version control (using GitHub or GitLab), code analysis and code coverage (using SonarQube and JaCoCo), and deployment to a binary repository server (using JFrog Artifactory). I have described this in detail in the document Continuous Integration with Jenkins.
As a cloud alternative to Jenkins, I would also consider GitHub, where Actions make it easy to automate workflows and to publish reports on GitHub Pages.
One of the important features of a CI server like Jenkins is that it automatically triggers the build, based on defined criteria. Jenkins can be deployed to set up an automated testing environment where Selenium WebDriver, Playwright, REST-assured (for REST API tests) and other tests can run unattended based on a defined schedule, or every time changes are submitted to the Source Control Management (SCM) tool. Test environments can be on local physical or virtual machines, or created on demand in the cloud, or provided on Docker containers.
All program code (including all unit and integration tests) should be checked by a code analysis tools as part of a Continuous Integration pipeline. I would use a code analysis and code coverage tool like SonarQube (including Codecov or JaCoCo). SonarQube ensures that all program code meets the quality criteria and Codecov (online tool) or JaCoCo (local install) ensures that all program code is covered by unit tests. JavaScript (Jasmine) code analysis can be achieved with open source tools like Istanbul.
Because version control systems such as Git store code and not binary files, a binary repository tool such as JFrog Artifactory should be integrated in the Continuous Integration pipeline to track builds, dependencies and deployment history. The following diagram depicts how a binary repository tool such as JFrog Artifactory works with Jenkins to store build artifacts.
For mobile testing, I would use Appium, because it allows testing of all three types of mobile applications: native, hybrid and mobile web. It executes tests on actual devices, emulators and simulators. Most importantly, Appium can execute the same Selenium program code on both Google Android and Apple iOS, without the need to recompile apps, or having access to the source code.
For testing Google Android apps only, I would also consider Google’s free Android Studio IDE, which allows JUnit based User Interface tests using emulators or real devices. Android Studio also has a host of very smart, performance monitoring tools that allow developers to fine tune projects and maximize their efficiency.
I would also consider the use of Groovy for easier and faster writing of automated tests. Groovy extends Java with scripting capabilities, but it is completely optional in Java projects. It mixes and matches perfectly with existing Java program code and libraries, and it compiles straight to Java bytecode. Java projects can contain anything from zero percent to one hundred percent Groovy, so it is perfectly possible to start very small and to extend the Groovy portion over time, without the need for rewriting any existing Java program code.
If the team uses Android Studio for Android app development, then I would consider using the Kotlin programming language instead of Groovy. Kotlin is fully supported in Android Studio and is very similar to Groovy, but Kotlin is statically typed and has at least two major advantages over Groovy. The first major advantage is that Kotlin saves a lot of error-handling code by getting rid of null pointer exceptions, which the compiler achieves by not allowing a null value to be assigned to any object reference. The second major advantage is that Kotlin favours composition over inheritance, which allows for much cleaner and more flexible (Agile) design decisions than traditional Java inheritance.
I have both written and introduced test automation frameworks in multiple projects. My current framework for Selenium, Playwright, and REST-assured (for REST API testing) is described in the document Java Framework. The framework is based on Apache Maven, TestNG, and Docker. It can be driven from the command line and therefore integrates perfectly into CI pipelines. It can use test data from any data source, use multiple test reporters, run tests headless (without GUI), deal with proxy servers, execute multiple tests in parallel, and run any combination of test groups, such as regression or smoke tests. It also makes it easy to use cloud providers and Docker containers. Selenium and Playwright desktop, Appium mobile, and REST API tests can be executed in parallel from the same Java framework.
If the team practices Behaviour-Driven Development (BDD), then I would use the open source library Serenity BDD for reporting. Serenity BDD can be used both with Unit test tools (such as JUnit or TestNG) stand-alone, as well as on top of specialised BDD tools, such as Cucumber/Gherkin. Serenity BDD automatically generates attractive, graphically illustrated, narrative reports that document what the application does and how it works (“living documentation”). Serenity BDD tells not only what tests were executed, but more importantly, what requirements have been covered and tested. Serenity BDD can be integrated with requirements (user stories etc.) stored in an external source (such as Atlassian Jira), or just use a simple directory-based approach.
