Automated software testing plays an important role in ensuring quality at every stage of software development. This article describes how to get started with Python’s Robot Framework. […]
Any code change carries the risk of causing unwanted side effects that lead to an error state in the software. In order to find such errors at an early stage (and not just in production), automated tests must validate both the previous and the new functionality of the software.
First, the project goals (vision) and a rough project plan (roadmap) should be defined. The necessary budget should not be neglected. As expected, the effort for the development of fully automated tests at the beginning of the project is significantly higher than later in operation. However, a certain “background noise” always remains, as test results have to be evaluated, test cases adapted and new test functionalities have to be implemented.
After the procedures and processes of the software to be tested are clearly defined, the tools and frameworks are selected. The first prototypes were then to be presented in a kick-off in order to create the necessary acceptance for the tools used.
Automated Software Testing – Introduction
After successful creation of the (new) software, the test automation is automatically started with the software version to be tested. In the case of installable software, this must be done without manual intervention in the so-called “silent mode”. By the way, most software installers offer a “response file”. A response file is a text file that contains the corresponding dialogs and inputs that the user specified during the initial installation.
The software is then automatically set up and parameterized, and the correct operation of applications, services, browsers, GUI, etc. is tested. If the software product to be tested supports different platforms (Unix, Linux, Windows), all these operating systems must be tested, preferably in parallel and in a nightly run. In this way, long-running tests can be carried out efficiently, the utilization of hardware resources is better distributed, and the test results are available for further analysis in the morning.
If a large number of tests are running in parallel on several different test systems, it is advisable to use a virtual computer system, a virtual machine (VM). If you don’t want to use your own hardware, cloud computing is an interesting alternative. The required IT resources are also made available here.
Testing with the Robot framework
For the development of fully automated software tests, the Robot Framework is my clear favorite. Decisive factors are the extensive modular system for expanding the range of functions as well as the easy handling with good integration options. Another advantage is the simple, table-like structure of the test cases, which requires only a short training period.
The Robot Framework focuses on user testing (acceptance testing and end-to-end testing), but is also used for system testing and smoke testing. The Robot Framework developed in Python is freely available as open source software (Apache license). A new major release, Robot Framework 5.0, was released at the end of March 2022.
The installation of the framework is uncomplicated, since only Python 3 and its package manager “pip” are needed.
Two established systems are available for development and test execution, but they can also be used together to take advantage of the respective environment. RIDE is the robot’s own development environment (IDE), which makes it easy for newcomers to use the framework.
RED is an Eclipse-based development environment that will delight experienced software developers.
Test Automation Scenarios
As described above, all functions of the software should be tested automatically. The Robot Framework helps us with a variety of libraries and additional packages that can be easily integrated. An example is the very stable SSH library, which makes it possible to run all tests remotely and reusable on Windows, Linux and Unix.
In the area of end-to-end user Interface (UI) testing for web browsers, the new and modern browser library from Playwright (since the beginning of 2021) or the long-established Selenium framework are available. These libraries can be easily integrated into the Robot Framework. This makes it easy to test graphically based functions such as mouse clicks or user input and output on websites.
Of course, self-written test scripts can also be easily integrated into the Robot Framework. It does not matter whether Java, Python or shell scripts are preferred, only the return values of 0 or not 0 must be taken into account for meaningful reporting.
Software testing – Evaluation
Extensive test reports are automatically generated by the Robot Framework as HTML pages. In addition, the test results can be imported into a test management system such as TestLink. In order to be able to analyze errors faster, an exact error description with output of the log files is very helpful. For a large number of tests, it also makes sense to define the responsibilities for the tests. This makes it possible to address the corresponding colleagues with the error report via an automated e-mail.
The icing on the cake for the graphical presentation of the test results is the integration with e.g. a Splunk dashboard. This makes it easy to view the current situation, especially with a large number of software tests.
If the dashboard is not enough, a traffic light system is the best solution. With the previously defined thresholds, the different colors of a USB traffic light can be used.
For example, if you come to the office in the morning and see the traffic light shining red, it is immediately clear: errors have occurred. On the other hand, if you see a green traffic light, the day can start more relaxed, because all the tests have run and the software has passed without errors.
Especially in agile software development, fully automated tests are an indispensable factor to ensure continuously error-free software. Valuable resources can be saved by test automations that run at night. The Robot Framework is an excellent tool with which all software tests can be automated.
*Timo Füg is a software developer with a focus on software quality assurance. He works at Uniserv GmbH as Technical manager of software quality assurance.