Quality assurance (QA) is the central element of any software development lifecycle (SDLC).
Whether you develop a new solution or update an existing one, you need to test it for usability, performance, and security among other factors. The better your QA process is, the faster you can schedule new releases. That’s a well-known axiom — but a tough one to implement as market data shows.
Poor software quality in legacy systems is pegged to cost the US economy over $1.56 trillion per year — a 22% jump from the 2018 figure. Support of such systems costs a lot — and high error rates make any type of legacy modernization project more complex to execute. Software quality assurance, delayed to post-development, also builds up technical debt, forcing teams to tackle the reworks, instead of focusing on extra value-driving features.
Given that corporate technical estates are growing large every year, leaders need to come up with better approaches to quality assurance.
And one of the main methods for that is QA automation.
What is QA Automation?
QA automation assumes using tools to automatically review, assess, and validate a software product against the set requirements for code quality, functionality (business logic), and user experience.
In other words: you perform test automation and gain an automated testing experience as an output — just so that you are clear on the basic QA lingo. Instead of manually going through the standard testing sequences, quality assurance specialists created automated test scripts to probe the produced code for conformance with quality, security, or business requirements. Many of the performed tests are routine, repetitive step-by-step processes, required to validate the written code after changes or pre-deployment.
Given that organizations now do around 1,460 code deploys per year on average, over-reliance on manual testing means two things: larger headcount or slower time-to-market.
Automated software testing allows you to improve software testing speed without compromising its quality. A purpose-built automated testing framework can help you test all the nooks and crannies of your product and ensure end-to-end tests coverage for different usage scenarios.
A test automation framework is a collection of rules, tools, and processes your organization uses to perform comprehensive QA at different stages of your software development lifecycle.
Sample Test automation framework
Why Automate Quality Assurance
In 2011, Google engineers managed to develop and release over 100 Google+ features over a 100-day period. How did they do that? By strategically blending manual and automated testing at different stages of SDLC.
Since that time Google “upgraded” its baseline setup to Google’s Test Automation Platform (TAP) — the world’s largest software test automation framework, backed by DevOps principles such as Continuous Integration (CI), Continuous Testing (CT), and Continuous Delivery (CD).
On a regular day, TAP integrates and tests over 13K code projects, requiring 800K builds and 150 Million test runs.
The best part? Even on a smaller scale, test automation results in remarkable processing speed and some extra benefits such as:
- Immediate feedback on newly-shipped code
- Higher cost-effectiveness, compared to manual testing alone
- Rapid execution speed that improves teams performance pace
- Higher test accuracy since there’s less room for human error
- Better test coverage as you can do more faster
- Test case reusability, minimizing redundant work
- Faster time-to-market for new products
That being said, merely installing a set of QA test automation tools isn’t enough to get the above gains. You also need to consider how you will embed automated QA at different stages of your SDLC to be performed alongside development.
The “merger” of development and testing work is called continuous testing — an Agile practice of running automated tests as part of the software delivery pipeline to obtain feedback before moving on to the next stages.
CT is an integral part of Continuous Integration and Continuous Delivery (CI/CD) — a wider set of operational practices, aimed at bridging the “hand-on” gaps between different teams that take part in software development.
CI/CD and continuous testing are proven approaches to improving your software development cadence. But they are hard to implement, too. Per Capgemini 2020 Continous Testing report, businesses face challenges across several vectors:
- Process adoption: 56% find in-sprint testing hard to execute because of process constraints.
- Alignment: 67% also struggle to maintain meaningful test cases that align with user expectations.
- Talent: 62% can’t find enough QA automation engineers to implement a test automation framework for them
How to Build a Test Automation Framework Step-by-Step
Over 95% of organizations agree that test automation patterns, principles, and practices are critical for them. But far less made progress with test automation framework design and implementation.
Indeed, QA automation is a complex process, but a systematic one. Meaning you need to complete several core steps to transition from largely manual to mostly automated testing.
Our QA team unpacks the best practices below and shows how these are implemented in practice with a client case study.
1. Determine The Optimal Test Automation Framework Architecture
Quality assurance is a multi-facet activity. You can probe newly-built software from different angles to ensure it works like it was meant to and has no critical bugs or vulnerabilities.
For that reason, there are plenty of different types of tests:
Functional testing
- Unit tests
- Component tests
- Smoke tests
- Regression tests
- API tests
- UI tests
Non-functional testing
- Performance tests
- Load tests
- Pentesting (security tests)
- Stress tests
- Volume tests
- Recovery tests
Can you incorporate all of them into your AQA framework? Perhaps. Should you attempt to do that? Not at all.
As Denis Kolesnikov, Expert Test Automation Engineer at Edvantis strengthens:
“A good automated testing framework should focus on a specific problem(s) and be designed around your type of product. Before you start designing your architecture, you need to get your QA priorities straight. Put down your main testing requirements, select a series of tests you need, then look into different strategies for automating them”.
A test automation framework is a collection of process steps and guidelines for creating shared process automation for a standardized set of tests. Similar to other types of automated processes, test automation requires a shared business logic for design.
Various test automation frameworks exist, each with a different logic:
- Keyword-driven framework. This is a popular approach to automated functional testing. You break down the test logic into keywords and functions and store them in a separate Object Repository. Then call it in when you need to run a test.
- Modular-driven framework. In this case, you analyze all available test cases to identify repeatable flows. Then turn these reusable flows into functions and score them as test scripts. To perform different testing scenarios, you then re-use or combine these modular scripts.
- Behavior-driven framework. All application behaviors are defined through human-readable test commands (e.g. register an account). Then these are codified into test cases using BDD tools.
- Data-driven framework. You build your test cases around available test data, stored in a table or spreadsheet format. Then run a single test script to validate all the data and generate outputs in the same table.
- Hybrid framework — any combination of the above AQA frameworks.
“Each framework on the market is neither entirely good nor bad — everything depends on your particular project, solution, and software development methodology,” Denis Kolesnikov also notes.
2. Design a Reference Test Automation Framework
Conceptually, all QA automation frameworks have four layers:
- Test generation
- Test definition
- Test execution
- Test adaptation
Jointly, these govern the relationships between a TAS – test automation solution – and a SUT (system under test). Or, to put it another way, these four layers create the rules and guidelines for testing automation.
Test Generation Layer
The test generation layer sets the baseline for subsequent QA automation. Your goal here is to formalize your main test types, design manual tests, and set up test data capture and generation to support the selected QA scenarios.
The components (and supporting tools) of this layer are used to:
- Edit and navigate test suite structures
- Align test cases to test objectives (or SUT requirements)
- Document the test design
Test Definition Layer
At this stage, you need to analyze the available test data and match it to the planned test cases. Then begin designing test scripts for the approved use cases.
The components (and supporting tools) of this layer are used to:
- Partition (constrain), parameterize, or instantiate test data
- Document the test data, test cases, and/or test procedures
Test Execution Layer
With all the prep work done, you are ready to move to actual QA work. This part of a test framework includes all the tools you need to run, log, and report on automated test case execution.
A robust test execution layer provides the following capabilities:
- Auto-set up and tear down of the SUT for test execution (i.e. set of test cases including test data)
- Rapid configuration and parameterization of the test setup
- Interpretation of both test data and test cases and subsequent transformation of them into executable scripts
- SUT responses validation (i.e. comparison of expected and actual results) for reporting
- Precise control and proper timing for automated test execution
Don’t rush with implementing a definite architecture. Instead, start with several proof-of-concept (POC) scenarios. Overall, 20% of QA teams say that selecting the right test case scenarios is challenging for them. So there’s going to be some trial and error.
3. Decide on Your Test Automation Tech Stack
As mentioned earlier, a testing automation framework is a combination of processes and tools. The previous section described the process layout. Now let’s focus on the QA automation tools.
QA automation testing tools can be:
- Open-source – community-developed and supported software you can use without any license purchases. There are no constraints for customization.
- Proprietary tools – licensed software, offering access to a rich set of native functionality out of the box.
- Custom solutions – any combination of the above two options within your AQA framework.
When deciding on your QA automation stack, you should always take into account:
- Programming languages (most QA tools are language-specific)
- Operating systems (your testers are using)
- Platform support (web, mobile, tablet, etc)
Some of the automated testing tools we recommend at Edvantis are:
- Selenium WebDriver – cross-browser web testing framework
- Cypress – end-to-end JavaScript testing framework
- WinAppDriver – open-source desktop automation testing framework
If you need more options, NPM Trends lets you compare individual QA automation tools and test automation frameworks.
4. Decide on Your Path to Implementation
After you’ve selected your test framework, you need to decide on the implementation approach. To better illustrate different paths to implementation, we’ll use Selenium.
Similar to other QA automation tools, Selenium supports several test design patterns:
- Wrapper methods: In software engineering, a wrapper is a method of encapsulation. Essentially, you create a custom variant of executing a certain function because the original one doesn’t meet all your requirements (e.g. it is too complex to execute or doesn’t perform all aspects of testing you need). Then add this wrapper into your framework.
- Abstraction methods: Abstraction is a technique software engineers use to hide unnecessary details from the users. In test frameworks, abstracting certain functionalities promotes better code structure and maintainability.
- Page Object method: This is the most commonly used method for organizing test cases. Here you denote each web page of an app as a class file — such that contains only all corresponding web page elements. Then perform tests on these elements. Doing so reduces code duplication and improves test maintenance.
Again, no method is inherently better or worse — they are just better suited for different use cases. An experienced QA testing services provider can advise you on the optimal methods.
5. Develop Test Data
Automated testing is heavily-reliant on data. The more data you have — the better outcomes you’ll gain. However, when you are doing QA at the pre-production stages, you may be lacking meaningful first-hand data to cover all testing scenarios.
Therefore, consider extra test data generations strategies such as:
- Synthetic data from automated data generation tools
- Mass copy of data from production to testing environment
- Mass copy of test data from legacy systems
- Manual test data creation
Next, you should group all available data by test cases and securely store it in a test data management tool.
Finally, remember that QA automation is a continuous process. You’ll have to work in increments and progressively extend automated test coverage to new test scenarios. It always makes sense to start with several testing scenarios — and then add extra ones to your framework.
How We Helped An Enterprise Client Switch From Manual to Automation Testing: Case Study
Now let’s move from theory to practice. In this case study, we’ll show how we helped one of our clients, a software provider for oil & gas companies, progressively implement a QA automation framework.
The Need to Automate
When we began our collaboration, the customer had an in-house manual testing division. However, the team wasn’t coping with the growing workloads assigned to them. This led to a longer time-to-market for new software functionality and costly code reworks post-release.
The testing team spent a lot of time on regression testing in particular — a strong contender for automation. So the client approached Edvantis’ QA specialists to help with test automation.
Building a Custom Test Automation Framework
The company has a lot of JS applications (web + mobile). To ensure sufficient coverage, we selected WebdriverIO as the pillar element of our QA automation framework. Then augment it with extra customizations/integrations where needed.
Tools we used as part of the framework:
- Web:
- BrowserStack
- Selenoid
- Mobile:
- Appium
- Additional:
- Mocha
- Chai
- Babel
- Grunt
- Report Portal
The client was initially skeptical about the feasibility of automating certain test cases, but we successfully did:
- Mobile and tablet app testing (iOS, Android, APIs) automation
- UI testing automation
- API testing automation
During the course of the implementation, we’ve developed a secure repository for storing all test cases (plus supporting data), and a user-friendly portal for checking status reports and generating custom ones. Currently, we are working on extra Grafana dashboards — an application that would enable the company to collect better analytics on test pass/fail rates and track other metrics.
A Robust Test Automation Framework and Team Upskilling
At this time, we keep expanding the framework coverage. Up to date, we achieved 70%-80% automation rates on two major projects and 50% on the mobile test automation framework in particular.
Some projects are not ready for automation yet because of security/compliance reasons. So in the meantime, we focused on re-training the client’s manual testers.
In the course of the partnership Edvantis:
- Delivered a QA automation training course for a 10+ people team
- Helped fully re-train and re-assign two people to the QA automation testing unit
- Continues expanding the client’s automated test case database, coverage, team upskilling activities.
By choosing to work with our company, the client has not just obtained a new AQA framework and faster time-to-market but also benefited from knowledge sharing and cross-pollination.
To Conclude
In our experience, the optimal way to approach QA automation is to start with several test cases for one project, fully codify them into a reference framework, and then progressively add new test cases.
By working in increments, you get a chance to re-assess your testing cadence and locate gaps (or redundancies) in test cases designs. Doing so can reduce the testing speed without undermining test quality and help you bring more competitive digital products to the market.
Contact Edvantis to receive personalized consultation on QA automation implementation for your company.
