Why Dedicated Software Development Testing Matters

It’s not uncommon to hear clients on the business side question whether it’s really necessary to budget for dedicated testing resources as part of a software development project. “If the development team followed spec and developed according to plan, how much testing is there to be done?” Sounds like a valid question.

But when you take a closer look, this situation is quite similar to when executives ponder budget requests for additional staffing and technology in Marketing. Sure, we need Marketing to drive sales; but really, how many people and tools could they possible need to plan and execute? In reality, quality control management has always been critical to delivering high quality software; but what’s not always obvious is that it covers much more territory than simply exposing errors or bugs.

Comprehensive Testing Adds Value

Testing teams are dedicated to validating that a software program or app or product meets the technical requirements and design guidelines established at the start of the project. In addition to validating functionality, they also cover standard compliance, non-functional requirements, as well as integration with other systems. This broad approach improves the end-product quality and delivers a positive user experience.

That said quality control management should not become the “gold-standard”. Expert testing teams must be diligent about identifying and covering the “must-have” scenarios and cases – within project constraints.

In today’s market, there is the addition of complex Digital Transformation initiatives, where there are typically many connected moving parts that must function together seamlessly – multiple platforms and systems, devices, Websites, and eCommerce capabilities. Development work in a specific area can easily impact how the new software interacts with existing systems which makes testing – specifically Integration Testing – a critical part of the complete software development project.

Agile testers provide quick feedback on each point which makes fixes easier and less painful. Agile means open daily communication and continuous involvement of the testers as part of the development team instead of functioning as a separate team in a traditional plan-driven development approach. They not only function as testers, but as BAs and as UX experts.

A typical day in the life of an Agile tester is very busy with interaction, discussions – not only the function of testing. Agile embraces change during any stage which makes testing more, well, agile. In contrast, testing in a plan-driven development model leaves the majority of testing to be performed after the development work has been completed. Depending on the testing results, this approach could have the effect of delaying delivery especially if a high number of serious defects are identified.

Quality Control Structure and Testing Flow

We believe there are 3 main phases when it comes to testing:

1. Define the test strategy and testing plan
2. Execute on testing
3. Assess and Report on the readiness or “done” status

Phase 1 – Define the test strategy and testing plan

At a minimum, it’s important to have a vision for how testing will be executed throughout the development phase. Or, the team may create an initial version of the testing plan prior to the start of the project. The level of plan detail is dependent on customer expectations, the project lifecycle, and the size of the team. What matters is that there is a clear understanding of business and technical requirements to ensure they are covered in the testing plan.

This document should describe the testing goals and quality standards, hardware and software testing environment, tools, types of testing to be used for each project phase, and testing results reporting. The objective for this document is to define what level of testing quality will be acceptable and how the testing team will meet the requirements.

It is important to ensure the testing process is continuously updated with any changes such as changes to the focus or goal of testing, or the testing expectations. To ensure the effectiveness of the testing process, the test leader must be able to adapt to the project changes.

Phase 2 – Testing Execution

Test cases may be high-level and describe main steps, or they can be quite detailed, to the point that less-experienced testers could execute them without knowing all the details of the project. The level of detail required depends on the project needs and is a part of a negotiation between the Project Manager (PM), Business Analysts (BAs), and the Test Leader.

Test cases and scenarios are live documents and must be kept up to date. As requirements are updated, the related cases must be updated at the same time. The output from this activity will be functional test cases and scenarios.

Each test case should have a clear entry point, testing step(s), along with the expected results.  The creation of test cases is often dismissed as not important by individuals on the business side. But without clearly articulated and documented test cases, the steps of how requirements will be verified will only exist in the tester’s mind while repeatability is one of the main characteristic of a mature testing process.

Depending on the project lifecycle, test cases and scenarios may be created and tracked in a separate phase or this work may be completed along with development or even test-first approach.

Options for testing software solutions

The testing team may decide to use a powerful ALM (Application Lifecycle Management) Solution such as Microsoft TFS (Team Foundation Server) or QC (Quality Center) by HP, to track test cases and link them to the appropriate requirements. This is an ideal scenario since using an ALM tool provides traceability between requirements, test cases and discovered defects, making the current status clear and transparent for all stakeholders.

If the preferred, but more expensive tools are not available, the team may choose to use tools with less integration such as Zephyr (the testing support plug-in for Jira), or free tools like Test Link, or even smartly structured Excel documents. The goal is still the same – tracking and traceability.

Other key factors in testing execution

Specific test data

Specific test data may also be necessary for certain types of business scenarios. Producing test data is the responsibility of the testing team and requires technically skilled testers. When preparing long xml or json files, they need to be varied to cover different cases and then updated once related functionality is changed. At First Line Software our testers use Python, JavaScript, Powershell, or other widely used languages to optimize this time-consuming work.

Defects

Defects found during the test execution phase should be tracked in a proper system. It is ideal to use tools such as Jira, TFS, or tools of similar quality. Some customers provide their own testing environment and provide secure access to the First Line Software testing team. If a testing environment is not provided, the First Line Software team can use Redmine, which is a bug-tracking tool.

Impact Analysis

Impact analysis is another important area of testing. Skilled professionals identify the critical areas where testing efforts should be focused.

Regression Testing

Regression testing is often the last step in the Test Execution phase. After performing testing of new functionality throughout the development phase, identifying defects, and re-testing defects, the Testing team executes a final run to verify that all functionality which worked previously has not broken with the introduction of new functionality.

Test Automation

Test automation has always been a part of excellent testing practices.  Since test automation is expensive, especially at the start, it is important to understand why it is needed.  Test automation is used primarily for Regression Testing; but some projects support automation from the start of the project. Regression Testing can be UI-driven or pure backend and First Line Software testing experts are equipped to propose a solution.

A new challenge has arisen in Test Execution. Many projects require short development phases and frequent releases. Clearly, a project with one-week sprints cannot afford to spend even one full team day on Regression Testing. Test automation can be a part of the solution; but the team must clearly understand what to automate and what not automate. Automating at least several frequently-used scenarios not only saves time; it also provides the ability to introduce changes later-on during the sprint.

Eliminating Testing from a development project can have serious consequences in terms of achieving the desired quality of the end product and duration of the project. Whether your company is outsourcing the entire development project or simply the testing phase, be sure you allocate budget for this important function.

Phase 3: Assess and Report Readiness

Testing teams must always address these three questions:

• Have all requirements been met?
• What is the current status?
• Is the product ready to be released?

To answer these questions, the testing team should employ the use of smart metrics. These metrics can include testing statistics such as a number of attempted cases, passed/failed, and product statistics which include the number of open defects, defects by status, defect density per line of code, and so forth.

Metrics is a powerful tool for testing project management. First Line Software typically recommends using a default template for test product metrics, which can be customized according to the specific project.

When it comes to readiness reporting, it is not enough to simply list covered features and related defects. While it is possible to deliver working software with major bugs; what matters most is to ensure the Product Owner understands the defects so they can make an informed release decision. First Line includes an Executive Summary in the Readiness Report with recommendations for the development team and client about what is critical to address prior to the product release.

How Expert Testers Can Contribute to Testing Success

Testers need to understand the project goal, the business itself, and the requirements for the product being built. They should have some BA skills or work with BAs and the Project Manager as a team, to select the appropriate quality criteria.

In an Agile environment, testers analyze the scope together with the development team plus a BA, Product Owner (PO), or a PO deputy, to ensure the scope is transparent, the testing goals are understood and the team is on the same page. With this approach, any questions can be addressed immediately. Finally, the team agrees on the acceptance criteria and works out the common goals.

Testers should also have a solid technical background to ensure they understand the project technology and architecture – at least on a high level. Testers who meet the criteria will be able to tell which part of the product is being developed and what is not considered a part of the development project. A skilled tester will also be able to identify the potentially fragile areas and accordingly, place a greater focus on them.

Key Tester Skills and Attributes

• Expert testers should possess a wide range of skills including solid analytical and technical skills.
• Collaboration skills – specifically communication skills – are crucial for agile, especially with virtual and international teams. These skills will be utilized throughout the project.
• Testers need to be continuously looking for how they can improve the quality of testing and the testing process, and be able and willing to use the best tools for efficient testing.
• Agile testers should not rely solely on specifications since they may not be clear or in some cases, not exist at all. They should also be proactive about looking deeper and applying best practices and trends in Testing.
• Agile testers must react promptly and have the ability to identify “must-have” fixes from “fixes that can wait”.

Types of Software Testing

A range of testing types may be used to ensure product conformance with requirements. Depending on the Systems Development Lifecycle (SDLC) and project goal, testing can be manual or automated. The most common types of testing are:

Static Testing and Sanity Testing

Used primarily to verify that requirements are clear and testable.

Functional Testing

Functionality is at the core of the application. Various practices are used to make functional testing effective. For example, some projects use a Transition procedure that enforces communication between developers and testers to help identify the most obvious issues early on. Agile projects use Sprint Planning during which developers and testers discuss functionality before starting development, clarify all details and ensure final transparency for expected results.

Load Testing

This non-functional testing is focused on product behavior under various load levels. For example, we use the Load Impact tool to create scripts that emulate large numbers of simultaneous users opening different web pages. The groups are distributed to a number of geo-locations and user behavior is scripted using Lua language. The testers create scripts, define the load level and schedule the scripts. While the scripts are running, the team monitors the back-end using server agents and collects key metrics such as CPU usage, Memory usage, network, etc.  

UI and UX Testing

With the vast number of mobile devices in use and frequently changing browser specifications, this testing is important for identifying UI issues that will negatively impact the user experience.

Integration and System Testing

This type of testing is extremely important for complex projects. Once the components are tested, the connectivity between the components needs to be verified. System testing means testing software as a whole, not part by part. 

Regression Testing

Regression testing is more about when and what to test rather than how. There are times when testers need to verify that the product is free from degradation, which can be introduced as a side-effect of recent changes.

Smoke Test

Similar to Regression Testing, it is not about how, but about “what”. The Smoke test is a small scenario aimed at checking if the application is ready for deeper testing and is often also used for products with frequent releases – after deep functional testing – as the final activity to make a Go-No Go decision. A Smoke test needs to be focused on what is most important to the specific development stage.

Mobile testing

Testing on mobile devices is a must-have for most, if not all, development projects these days. Whether it is a native app or a site with mobile support, it is important for testers to keep in mind the list of mobile-specific cases such as screen rotation, low Wifi, UI guidelines, etc.

Software Testing Tools

It is critical to use the right tools for the best testing results. The following list provides examples of testing tools used for specific types of testing by First Line testing experts:

• Test Case Management – TestLink, Zephyr, a Jira plugin, MS MTM
• Bug Tracker – Jira, TFS, Redmine
• UI Testing and Script Verification – Firebug, YSlow
• Load Testing – Load Impact, JMeter
• Functional Test Automation – Selenium, Ranorex, TestComplete, and Coded UI Tests (CUITs) for Microsoft

There’s no doubt Software Development Testing is a comprehensive topic and one that must be given adequate attention and consideration before launching any development project. We hope this post has been convincing about how critical Testing is to the success of software development projects.

Talk to Us!

Whether you have a complete software development project in the works or are seeking a project partner with Testing expertise, we invite you to contact First Line Software for an exploratory discussion about your development and/or testing requirements. Talk to Us today!