XenoGuard Documentation

Approaches to Test Automation

Test automation was the original application domain of XenoGuard but, due to its high practicality, our tool was discovered by clients from other industries.

Test automation deserves a separate chapter since the scope of its benefits is huge but requires a careful planning and a clear understanding of what you want to achieve and how.

Tests as Special Cases of Workflows

Test automation is very similar to workflow automation. You have a sequence of tasks to be performed by different applications and/or systems. You use a program (application or code script) that accesses these applications and systems and operates them to run such a sequence. 

The major difference from the workflow automation is that the goal here is to catch errors: scenarios when the tested applications fail to complete the tasks or throw wrong or unexpected results. 

Another difference is that test automation is used in (software) engineering and rarely by end users. Automated application tests play a big role in delivering high-quality software that satisfies user needs and works seamlessly with other applications and systems. The delivery must also flow uninterrupted. This became known as the concept of continuous integration and continuous delivery (CI/CD). 

Goals of Automated Tests

As a part of CI/CD, test automation gradually evolved into an independent field with a few established approaches, called test automation frameworks. Each framework is a summary of methods that derive from the goal of the testing.

The goals, as listed below, may overlap but, basically, they aim to ensure that:

  • functional requirements are implemented correctly
  • different applications are integrated properly
  • the graphical user interface works
  • the command line interface works
  • the application's performance is acceptable 
  • newly added features do not ruin any of the above

Test Automation Frameworks

There are different taxonomies that help you in understanding existing frameworks since the latter may overlap, too. The taxonomies are based on the two main questions:

  • Are you testing only single/standalone features or the application in its entirety?
  • How much flexibility do you need for your tests?

From Wholesale to Level-Based Tests

Originally, all automated tests were hard-coded. An automated test could only check one sequence of tasks at once and could not be re-used. That was known as the linear automation framework. Obviously, it soon proved to be inefficient. To address the issues, testers started to divide applications into modules and test them separately. The new approach – the modular test framework – allowed to re-use at least some parts of the script. But this method requires an additional step – integration testing – at the end of each test cycle to ensure that all parts that were tested independently also can work together as a whole.

Currently, software tests happen on four modular levels:

  • units
  • integration
  • system
  • acceptance

At first, developers and engineers test units: independent pieces of software. On the next level, another test checks if these units can work together. These are integration tests. Whereas integration testing still handles units as separate entities that only talk to each other, for the next stage – system tests –  the application is already "assembled" and must prove to work smoothly as a whole. Acceptance tests ensure that the application satisfies the requirements that were listed in the original specification and that the application meets the expectations of future users.

Systematics and Variation

The modular test framework posed a new challenge: how do you track what was already tested and in which combination?

Soon, two more approaches emerged in response to a need for a systematic and flexible testing: data-driven and keyword-driven test automation frameworks. One can even say that the keyword-driven approach is a special case of the data-driven one.

Data-driven testing means that your test scripts allow variations in the main scenario. The values that define those variations are stored in a separate file. For instance, the main script says "click on a button []" where square brackets mark possible variations. The data file would contain a table with values, such as A, B, C, or whatever buttons your tested application has. The script will access the supplementary data from the file and use different values – A, then B, then C, etc. – to click different buttons in different scenarios. This allows performing multiple tests in one run, without re-writing and re-starting the test manually.

With this approach, you can easily add new variations into the data file without changing the script. You can log test results and match errors with certain values to find out what makes your application fail. 

White-Box vs Black-Box Testing

Another taxonomy that exists in parallel with the previously mentioned ones, is white-box vs black-box testing.

Black-box testing considers an application as a closed entity and does not look into it. This kind of test does not check how the application works inside. It only examines if the application responds properly to external stimuli. For instance, if you ask a calculator application to put 5 and 3 together, it should throw 8. And it does not matter how it comes to this result. It is enough that it always generates the correct answer. 

Black-box testing examines the application against the requirements specification. It is usually a huge document that describes what the application should be able to do and how its interface must look like.

With the white-box testing, you look inside the application. Indeed, you also want it to generate some expected results in response to some external input, as in the previous case. The difference here is that you want the application to produce the result in the most efficient way. Thus, you check how the internal parts of the application work together.

Off-the-shelf, XenoGuard is a perfect tool for the black-box testing. But you are not restricted to that. White-box tests are possible for applications that have an API. On the side of XenoGuard, you will need to create a custom extension in C# using the .NET framework. 

This section may have given you a mixed impression: automated tests are powerful but you need to keep in mind a lot of things. Still, at MycroTools, we do believe that automated tests have a lot of benefits. We described them for you in the next section.