In this video, Alexander Meshkov explores some intriguing approaches to automation testing: Behavior-Driven Testing (BDT) and Model-Based Testing (MBT). Despite their potential to add significant value to the testing process, these methodologies are underutilized, with fewer than 10% of companies implementing them effectively.
Alex recently gave a talk on this topic, and you can follow along and view his slides here on Youtube.
Today I’d like to talk about how you can evolve your automation testing process by implementing BDT and MBT, especially the latter’s ability to boost the capacity of a testing team.
However, MBT doesn’t frequently get used in teams, and we’re going to explore why that is.
With over a decade of Quality Assurance (QA) experience behind me, I’ve been fortunate to lead and contribute to a variety of dynamic and challenging projects. Now I work for First Line Software as Leader of QA Practice and occasionally speak at different conferences. I like to talk about the optimization of the testing process, implementation of state-of-the-art approaches, and also about management in QA.
Let’s start with the history of testing. The first mention of software testing was in 1958 by Gerald M. Weinberg who created the first software testing team. For much of the 20th century, software testing was mostly manual. The first mention of automation testing was in 1999. It was the first book on software automation was written by Mark Fewster and Dorothy Graham, in which they comprehensively treated software test automation issues, strategies, and tactics.
Since the 21st century, automation testing has evolved very quickly, spurred by the creation of Agile methodology and DevOps practices. Today, automation testing is no longer new, but a fundamental component of software testing. However, it’s worth noting that 50% of companies are still at the second level of automation testing. The third level introduces Keyword-driven testing that can help facilitate the transfer of your automation process to Behavior-driven testing. The next level incorporates specific model approaches; and at the top of this pyramid is Automation Test Design. In other words, this represents a comprehensive automation testing process, from test design to execution. The two highest levels of this pyramid are what we refer to as Model Based Testing (MBT), which we will explore shortly.
As I mentioned earlier, many companies have yet to establish robust automation testing processes. The World Quality Report 2023 highlights the main reasons for this.
The first point from this report is that automation testing is often organized as a separate activity, isolated from the rest of the development and testing team in the automation process. I’ve seen this scenario in many projects, and I suppose that it happens because managers think that automation testing is merely an additional task for the development process. This segregation is a common mistake.
The second point tells us that teams often forget about the primary goal of automation testing, including the neglect to formulate a comprehensive automation strategy that incorporates specific approaches to testing.
It’s noteworthy that there isn’t more of an emphasis on the ROI of automation testing in the Report. This point is only in the fourth or fifth position. And of course, you often hear the complaint that our automation testing scripts don’t function correctly or that we can’t rely on them, which is why companies frequently revert to manual testing.
So, the question remains – why do these challenges persist?
If you look at the graph of a typical automation testing project, you can see that the more tests you have, the less time you have to create new ones without adding new automation testers. This becomes a question of budget, and there’s often a reluctance to expand the budget for these additional resources.
Typical challenges in automation testing usually stem from the following:
Also, another concern is the collaboration between the development team, QA manual testers, and automation testers. Sometimes, automation testers work in isolation from the rest of the team, and automation testing isn’t integrated into the CI/CD process. Manual testers and development teams don’t use automation tests in their work, leading to a situation where automation activities are done for their own sake, not for enhancing the development process or speeding up QA stages. It’s unfortunate, but there are methods to address this and achieve a more effective automation testing process.
To kickstart the transformation, we must begin by changing habits to incorporate QA practices that can enhance the QA process. So we’re going to explore the general approaches to automation testing and see how they work in practice.
If you have been working with script-based testing, you might find your automation tests becoming complicated and cumbersome. The first step to streamline this process is implementing Keyword Driven Testing (KDT).
Keyword-driven testing is a scripting technique that uses data files or tables containing keywords relevant to the application being tested. These keywords are the basic functional sub-procedures for the test cases of the application under test. A test case consists of one or more keywords.
To put it simply, the keyword is like a building block that we can use to create automated tests without the need for new scripts. The accompanying images [see video] provide simple examples of a login sequence and how this sequence can be used in the test case for creating an order.
To start, you need to segment your test cases into different blocks (items, functions, and sequences). Then, write separate code for each block. Subsequently, you can create various test cases using only the automation keywords. If you see a new keyword, you can add it to your automation testing backlog and address it in the next sprint.
It’s important to note that KDT will work only with full integration and collaboration between manual and autotest experts. Without the support of manual testers, this approach won’t work properly and your backlog of keywords could expand more rapidly than your automation capabilities. That’s why I prefer to assign clear responsibilities: manual testers should be responsible for defining keywords, creating test cases, and executing them. Automation experts should focus on coding, such as writing code for new keywords and developing features for frameworks and tests. This division of labor is the most effective way to achieve rapid and successful results.
KDT offers numerous benefits:
When you implement KDT, you can take your testing process a step further by integrating a BDD Framework into your team. At this point, you should start to convert your test cases into Gherkin language. I know that this is a very popular approach, so I won’t dwell on it too much. I’ll just mention that Gherkin language can help you enhance QA involvement, especially for those without coding skills, in the automation testing process.
Learn more: QA outsourcing or in-house testing? How to make the right decision
The typical QA process for BDT is excellent for Agile teams. As you can see, you can create feature files promptly after creating a user story. In my previous role, we refined our process to the point where we abandoned traditional test cases with steps and current and expected results. Instead, we taught our QA team to create test cases in the Integrated Development Environment (IDE) using only the Gherkin language. We didn’t write classic tests at all. To manage test case storage, we created a plugin that automatically migrated all Gherkin descriptions of tests from IDE to our Quality Management System (QMS).
I know that this process can work very efficiently. For instance, we managed to reduce the Time to Market (TTM) of testing by 4 times.
Finally, I would like to share some tips that contribute to a successful transition:
Let’s move forward to the next level of automation testing: Model-Based Testing (MBT). Is anyone familiar with the model approach and how it works? You’re probably aware that traditional testing constructs direct, forward-moving test cases. For instance, you might have a test case with six steps, which you test sequentially. Ultimately, at the end of the test, you determine a pass or fail outcome.
But let’s set aside this traditional approach.
Model-Based Testing (MBT) represents a fundamentally different approach, which is why it can sometimes appear complex.
Imagine you have a sequence of steps similar to the one mentioned in the previous slides. But you know that your system has branches, leading to additional test cases that you would typically write in a direct, forward manner if you continue to use BDT or classic testing. But if you take a look at the business process of your system, you will see that there are numerous potential pathways for your application to function.
The core concept of MBT is viewing your system as a state machine. A state machine model is a mathematical model that groups together all possible occurrences within a system, referred to as “states.” Every possible state of a system is evaluated, showing all possible interactions between subjects and objects. These are represented using graphs, where the nodes represent states and arrows indicate transitions between these states.
Here, you can see the state graph for the authorization process. You have different states like login_page, resetpassword_page, and more. Between these states, you can see transitions and events that must occur before moving from one state to another. This graph forms your test model of the System Under Test (SUT).
The distinctive feature of MBT is that you primarily need to describe the model states. There’s no need to write test cases or anything similar. If you’re familiar with test design techniques from QA training, you might recognize that the state diagram looks like the State Transition testing technique. So a solid understanding of test design techniques is crucial for implementing MBT effectively.
Once you’ve constructed the test model, an MBT framework, like the popular GraphWalker, will automatically create cases for testing your application. Navigating through the states is managed by a special algorithm, like genetic algorithms, which are equipped with distinct functions to run test cases. Of course, if you need to change your bypass algorithm, you have the option to adjust it in GraphWalker’s settings.
A critical aspect of creating a state model is to avoid loops in the process. There should always be a clear entry into the process and exit point.
Let’s discern the difference between BDT and MBT in the context of test design. Imagine your application introduces a new feature that adds a new transition between states D and F. Under a classic testing approach, you might generate around 4 test cases that directly examine the new branch from different starting points, such as AB and AC. But if you will use MBT you can essentially assure maximum coverage of your application’s pathways.
If you need to optimize the bypass algorithm you can add some restrictions to the framework. However, it’s important to recognize that MBT inherently allows broader coverage of your application without the need for extensive human resources. The primary role of test analysts is to construct a graph of state transitions. Then, your automation experts write code for each transaction. In the end, you get a functional graph capable of autonomously generating test cases.
The typical stack and process of MBT look like this: As input for the testing process, you usually have a user story or technical requirements. After that, a test analyst can deconstruct the requirements and delineate the states and transitions of the application. The graph is usually created in yEd, an application that has a user-friendly interface that can help you create a graph. These graphs can then be exported to Graphwalker in .ghraphml format.
While there are other applications similar to Graphwalker, such as FMBT or Matello, I wouldn’t recommend using them due to their less intuitive nature. Graphwalker has the ability to integrate with classic automation testing frameworks like Selenium, Appium, and Espresso. This integration allows it to identify and interact with the objects in your application and run autotests. If you require additional functionality, GraphWalker offers the opportunity to develop plugins or auxiliary tools as needed.
Finally, I want to show you an example of how you can elevate your MBT to automated test design. I came across a company with an internet application that devised an interesting solution. As you know, your customers are an invaluable resource for understanding what you need to test, since the goal of testing is to provide high quality for them. Customers may use your application in diverse ways, so their experiences can be instrumental in your testing process.
In this company, QA specialists created an application that converted statistics and customer usage patterns from Google Analytics into graphs. They would then analyze these graphs and make any necessary corrections – and that’s it.
Besides Google Analytics, tools like Splunk and other suitable applications can be employed to collect customer activity data. All you need are skilled individuals capable of creating a program to transform this data. From what I understand, the team mentioned above spent about 4 weeks on this task. I think that sounds very impressive.
Finally, I would like to compare the approaches I discussed throughout my presentation, highlighting the pros and cons of each.
First of all, it’s crucial to understand the nature of your business processes, If your application’s business process branches out at the beginning but then proceeds linearly, akin to an octopus’s limbs, without looping back or intersecting with other parallel processes, I advise you to implement BDT exclusively. You could try to implement MBT, but it’s unlikely to offer additional value over BDT, and it will demand more effort.
But if you have, for example, a mobile or internet application and your process can be depicted as a graph, implementing MBT can significantly reduce your codebase and maintenance time by over 60% compared to BDT.
Next, I want to draw your attention to the importance of collaboration, at least between QA Manual and Automation testers. It’s essential to delineate responsibilities clearly. Find a test analyst from the manual team capable of crafting Gherkin tests or graphs, educating other manual testers, and reviewing autotests. Automation testers, on the other hand, should concentrate solely on coding new methods for test cases and system objects, as well as enhancing or customizing the framework and developing new features and plugins to assist manual testers. This division of labor is key to achieving significant results in automation testing.
Finally, you need to be prepared to change your mindset and that of your team. It isn’t easy, but it is achievable. You might need to become an influencer within your company and persuade your team of the long-term benefits of implementing BDT or MBT. But in the end, these methodologies accelerate your testing process and increase the throughput of your team.
Now, it’s up to you. Thank you for your attention.
Learn How Our QA Services Can Benefit You
Alexander Meshkov is QA Delivery Director at FLS. Alexander has over 10 years of experience in software testing, organization of the testing process, and test management. A frequent attendee and speaker of diverse testing conferences, actively engages in discussions and keeps up-to-date with the latest trends and advancements in the field.
Whether you have a problem that needs solving or a great idea you’d like to explore, our team is always on hand to help you.
First Line Software is a premier provider of software engineering, software enablement, and digital transformation services. Headquartered in Cambridge, Massachusetts, the global staff of 450 technical experts serve clients across North America, Europe, Asia, and Australia.
Clinovera