Transforming Testing: A Recipe for Quality Engineering

By Tariq King, Ph.D., Director of Quality Engineering, Ultimate Software

Tariq King, Ph.D., Director of Quality Engineering, Ultimate Software

At some point during the evolution of the software engineering discipline, testing became the focus instead o f overall quality. Perhaps it was during the 1980s, a period characterized by rigid waterfall methodologies and heavy documentation. Fast forward to 2017 and several organizations are still struggling to recover from that heavyweight era, especially when it comes to testing. Over the last decade, there have been buzz words like “agile testing”, “continuous testing”, “shift-left testing”, and “quality engineering”. Although each of these terms has a slightly different meaning depending on the context in which it is used, they all share a common theme–quality should be an upfront, integral, and ongoing part of the software development lifecycle.

Quality can be defined as value delivered to a stakeholder, or set of stakeholders, at a given cost. Software quality can be achieved in multiple ways, but most techniques fall under two broad categories: fault prevention, and fault detection. Fault prevention techniques seek to avoid the mistakes that can lead to incorrectness, incompleteness, and inconsistency in the software being developed. They include following and applying good software engineering principles, practices, and processes. In contrast, fault detection techniques attempt to identify defects that exist in the product.

Testing is a fault detection technique where as quality engineering includes fault prevention, fault detection and more. Quality engineering encompasses broader ideas around what it takes to develop and deliver a quality product. For example, customer perception is a critical part of quality and must be carefully managed. You may build a highly appealing product that is feature-rich, user-friendly, performant, secure, and reliable, but if customers have poor experiences during activation, servicing, or support, it can leave them with a negative perception of quality. That perception is their reality, and their reality can cost you existing and future business deals. So how can an organization be transformed from one that thinks about testing as a phase after coding, to one that embraces quality throughout the entire lifecycle?

"Quality engineering encompasses broader ideas around what it takes to develop and deliver a quality product"

This is a recipe formulated from my experiences driving testing transformations in large scale software development organizations. It is aimed at provoking thought, promoting discussion, and encouraging others to share recipes that have worked for themselves. As with any recipe, before you start you’ll have to gather the right ingredients and necessary equipment. However, I must warn you that for this recipe good ingredients can be hard to come by, and some may even have to be made from scratch. On the bright side, most of the equipment is generally available and ready to use.

Ingredients

o A Culture of Quality
o Two Strands of Quality Teams
o Three Batches of Quality Engineers
o Four Layers of Quality Leadership

Equipment

o Organization
o Project Timeline
o Learning Infrastructure
o Product Lifecycle

“A culture of quality is a work environment in which people are motivated, empowered, equipped, and expected to deliver high quality software.”

The culture of quality ingredient must often be made from scratch and can be difficult to prepare. Influencing factors include executive buy-in on quality goals, access to learning infrastructure, and the organizational culture. Executive buy-in helps employees feel empowered to speak up and act on quality issues before and when they arise. There is nothing more motivating for the quality advocates working on the frontlines than hearing the executives communicate the importance of delivering a quality product. Maintaining a culture of quality requires access to the applications, services, networks, and people needed for enterprise learning operations. Learning infrastructure provides the tools necessary to drive the dissemination of best practices, quality goals, and other quality-related initiatives. Lastly, organizational cultures that emphasize the well-being and importance of people are highly amenable to the establishment of a culture of quality. This is because developing and delivering software with quality in mind yields several benefits to an organization’s people. High quality leads to less rework for team members, an improved experience for customers, and higher stock returns for shareholders. As part of the culture, it is also important to have a clear and shared definition of what quality is, and what it means to the organization.

“Having two strands of quality engineering teams can be highly effective in checking for a large breadth and depth of quality issues.”

One question that I am frequently asked is whether quality engineers should be integrated within development teams, or on a separate dedicated team. My general response is: “Why not both?”. In my opinion these two styles are complementary rather than competing approaches. An analogy I use is that the medical field consists of general practitioners and specialists. The former covers a breadth of factors when examining a person’s health, while the latter provides depth in a specific area. Similarly, a team of quality engineers can be spread throughout cross-functional development teams to perform general health and quality checks, while dedicated teams of quality engineers are leveraged to conduct deep specialized testing in an area of expertise.

“Skilled quality engineers are a key ingredient but they can be difficult to find. What makes it even harder is that you’ll want to get at least three different flavors.”

Quality engineers generally come in three flavors: functional, technical, and non-functional. Functional quality engineers focus on validating the end-user and domain-facing aspects of the software. They are good at exploring and validating the software from an external point of view. T his f lavoris w ell-versed i n b lack box, exploratory, and domain testing techniques. In contrast, technical quality engineers concentrate on technology-facing attributes. They have enough knowledge of programming languages, technology stacks, software architectures, and service delivery platforms to test the software from an internal perspective. This flavor is skilled in white box testing, development best practices, information technology operations, and full stack test automation. The third flavor of quality engineers are experts on validating one or more non-functional properties of the system, e.g., performance, security, accessibility.

“Investing in four layers of quality engineering leadership as your organization scales is an effective way of keeping the other ingredients fresh.”

Leaders develop and inspire their people, create clarity and alignment, and help the organization move forward together as a collective unit. As your organization grows, it will become increasingly difficult to maintain a culture of quality, cohesive quality engineering teams, and skilled quality engineers without good leadership in place. Depending on your needs you may want to invest in up to four layers of quality engineering leadership. These layers are leads, architects, managers, and directors of quality engineering. Quality leads work alongside quality engineers to promote and enforce best practices. Quality architects define the overall quality and testing strategies, mentor the quality leads and engineers, and drive the development and evaluation of tools to support validation and verification activities. Quality managers make tactical decisions for achieving quality goals, coordinate the day-to-day activities of the quality leads and engineers, and perform human resource functions for the team. The overall vision and strategic direction for quality is set by directors, senior directors, or a vice president of quality engineering. Leadership at this level interfaces with other departments, customers, and the professional community to drive quality initiatives.

“Once you have all the ingredients and equipment in place, the directions for transforming testing into quality engineering are relatively easy.”

Directions:

1. Spread the culture of quality throughout the organization. Transformation usually succeeds or fails because of people. Start by evangelizing and emphasizing quality as a key goal and responsibility of each person, team, and the entire organization.

2. Shift testing to the left on the project timeline. Facilitate earlier defect detection by training functional testers how to validate requirements, and pairing them up with business analysts. Similarly, shift your technical testers to the left by allowing them to pair and cross-train with developers to catch bugs as soon as they happen.

3. Use both strands of teams to blend quality into the product lifecycle. Long-term learning comes through practice and repetition. Leverage both types of quality engineering teams to reinforce quality goals and practices at every stage of the product lifecycle. Be careful to not let specialized teams like performance, security, and user experience slip into waterfall testing mode.

4. Place leadership layers on top according to the size of the organization. Whether you need quality leads, architects, managers, directors, and a vice-president of quality engineering, depends on the size, structure, and leadership needs of your organization. However, in general you’ll want to find a balance of leadership and engineering roles that works for you as your organization grows.