Business Vision on automated functional tests

INTRODUCTION

Automated functional testing is gradually becoming an industry standard as web and mobile applications are getting more and more complexe with ever-increasing challenges. It is therefore paramount to ensure applications are thoroughly tested to guarantee today’s standard.

Several approaches & solutions are available depending on the needs, the technical environments and the resources available.

Objective of this article is to introduce, from a business perspective,  the different types of automated functional testing in order to better address the business needs and facilitate the rollouts of such tests within the organisation.

TYPES OF AUTOMATED FUNCTIONAL TESTS

Before starting, a summary of the existing types of tests:

THE MAIN BUSINESS NEEDS

QUALITY & COVERAGE

Functional testing goal is to guarantee a minimum level of quality before deploying an application. By using an automated testing framework, we ensure that key  features of the application are systematically checked before deploying to production. This approach is also known as regression testing. 

In order to determine which key features are to be tested (aka functional scope), the below criteria should be considered :

• Business priorities : New tests should be carefully chosen and aligned with business priorities (e.g. what is the financial loss if the new feature breaks)
• Business expectations : the output of the tests should be aligned with the business (aka acceptance criteria)
• Tests duration : the whole duration of the tests should be manageable and should not slow down the release process. Ensure the core features are included in order to guarantee the application stability
• Tests complexity & maintainability : what is the technical feasibility to implement, deploy and maintain such tests. Will the tests require manual steps 
• Continuous integration : how do the tests integrate with the CI / CD framework ? Can the tests be triggered automatically ?

REPORTING

The outcome of test execution should be captured in a report showcasing successessful test scenarios and errors encountered. For errors, the elements necessary to analyze, reproduce and correct the error must be available.

ESCALATION PROCESS

As errors are encountered, it’s necessary to warn the key stakeholders (e.g. by triggering an alert) in order to analyze, prioritize and correct them before deployment. Depending on the organization, this can go through email notifications, ticketing system, instant messaging, etc…

It is important, the business should make the decision as to what should be next steps (eg. fix, accept, rollback, etc…) based on the impact & priorities of the uncovered issues. The existing process should be flexible enough to anticipate the possible choices. Several scenarios can be considered depending on the type of error encountered. It is good practice to have an escalation matrix (with impacts, priorities and next steps) available on-hand in order to be reactive and coherent about the decision making process.

DEFINITION OF FUNCTIONAL TEST SCENARIOS

Two groups of key scenarios to test :

• Structural components such as the header, navigation elements and the footer. They are essential for proper use of the site. It’s important to check their composition.
• Critical scenarios such as authentication, registration, product search, adding products to the basket, checkout …

LIST OF KEY ELEMENTS TO KEEP IN MIND

• Structural components
  • Home page layout
  • Product page layout
  • Navigation
  • Footer
  • Search result page (with & without results)
• E-commerce sites
  • Authentication (With or without SSO)
  • Account creation (With or without SSO)
  • Product Listing Page (PLP)
  • Product detail page (PDP)
  • Add to the basket
  • Backet handling (e.g. Modify quantities, delete items, … )
  • Guest checkout
  • Account checkout (initial & recurring  purchase)
• Editorial sites
  • Lead form
  • Application main features
  • Contact form

SCENARIOS COMPOSITION

Before being able to create the related tests, each scenario must be able to answer the following questions:

• What types of users (anonymous, logged, with specific access rights) ? This allows to know the preliminary stages of the test (ex: connection / disconnection)
• What is the test user journey ?
• What are the expected results for each step of the user journey ?

TIPS FOR WRITING FUNCTIONAL TESTS

Here are some tips for organizing and maintaining your tests:

• Factorisation : In a given journey, some steps are redundant with other tests scenarios (ex: search, add to the basket). It is therefore advisable to take these out in a separate test which will be called in due time.
• Keep user journey testing and page validation separate by respecting the following two principles: 
  • A user journey test should only check the elements necessary for the tested journey 
  • A page validation can verify the content of the page in detail. This test is independent from the others.
• Update existing tests scenarios with every new release in order to ensure they do not break. During development, developers must anticipate these impacts and update (or request to do) the test scripts accordingly.

TEST TOOLS

There are many tools available on the market. From standalone tools running locally on the developper’s PC, to complete SaaS solutions with additional  features such as application monitoring.

Here is a non-exhaustive list :

SIMPLE SOLUTIONS
Selenium IDE

• Format: Chrome and firefox extension
• Description of the tool: Selenium allows you to record the journeys and replay them locally on the developper’s PC
• Meets the following needs: Functional tests played locally in the browser
• Who: Business teams / project management
• Benefits : 
  • Writing quick tests
  • No technical skills required
• Drawbacks: Switching from local environment to server requires technical skills

CUSTOMIZED SOLUTION

Selenium (with Framework)

• Format: On server
• Description of the tool: Selenium (Framework) is a series of tools to control the browser and to validate assertions. Many frameworks have been built on it (Codeception, Cucumber, Junit, PHPUnit, Nightwatch, Behat, Behave…).
• Meets the following needs:
  • Tests integration in the development project
  • Integration with CI/CD processes
  • User journeys analysis
  • Can be combined with many plugins
• Who: Dev QA / System Administrator
• Benefits :
  • Very customizable
  • Integration into manageable CI/CD processes 
  • Easy integration with virtualization browser solutions (Browserstack, Sauce Labs…) for cross-browser testing
• Drawbacks:
  • Complex setup
  • Requires technical skills

END TO END SOLUTIONS (TESTS + MONITORING)

Asayer

• Format: SaaS solution
• Description of the tool: Devops tool for analyzing user journeys and their problems, identifying critical journeys and automating the corresponding functional tests.
• Meets the following needs:
  • Analysis of user journeys
  • Creation of automated journeys
  • Integration with some CI/CD tools
• Who :
  • Test management: QA / Business teams / project management
  • CI/CD integration: Administrator
• Benefits :
  • Integrates into the CI/CD process
  • No technical knowledge necessary to record the tests and play them individually
  • Integration with communication / reporting tools (Slack, Jira…)
  • Analyze customer journeys to identify critical journeys
• Drawbacks:
  • Adds a dependency on the front-end
  • Paid license

Testim.io

• Format: SaaS solution / Chrome extension (Freemium)
• Description of the tool: Marketing tool allowing to record & play  user journeys, and to connect them to CI/CD processes. 

• Meets the following needs:
  • Analysis of user journeys
  • Integration with CI/CD processes
• Who :
  • QA for writing tests
  • System admin for integration into CI/CD brick
• Advantages: Procedure for writing tests without code
• Drawbacks: Paid license

Leapwork

• Format: Executable (Windows | Mac OS)
• Description of the tool: Software allowing to automate actions and tests on all the applications accessible by the machine (web, desktop applications, software, VMs).
• Meets the following needs: Creation of automated journeys without technical knowledge

COMPLEMENTARY SOLUTIONS
Dareboost

• Format: SaaS (Freemium)
• Description of the tool: Dareboost is dedicated to performance testing and (synthetic) monitoring of web applications. Dareboost recently introduced user journeys as part of their complete suite of application performance monitoring tools 

CONCLUSION

I hope this article has helped you to have a more complete view of setting up functional automation testing. These will make your projects gain in quality while avoiding regressions and by saving time on revenue.