How Should I Organize My Automation Team?

By: Paul Grizzaffi

Joan Rivers once wrote, “I don’t want to order dinner by yelling into a plastic clown’s mouth”. We’ll get back to that shortly.

There is a compelling discussion to have about creating a dedicated team to “do all the automation”. Why not have all the automation developers or SDETs be on one team? After all, automation creation is a special skill like database administration and CI/CD/CT pipeline management, so shouldn’t we treat automators for testing in the same way? We can just hand them the requirements or acceptance criteria and they can bang out those automation scripts for us.

Some call the above situation the automation drive-thru; it’s an apt analogy. Like a fast-food drive-thru, you simply decide what you want to order, yell your order into a clown’s mouth, pay some fee, then get your food from the window. No muss, no fuss. Can’t automation development be similar? Instead of a greasy burger, some chicken nuggets, or some delicious tacos, out comes an automated test script; again, no muss, no fuss, right? Sadly, it doesn’t always work out with no muss or fuss.

The drive-thru is but one way to organize teams for automation; there are, however, three ways: the dedicated automation team, the distributed automation team, and the hybrid approach.

Let’s explore!

What Does A Dedicated Automation Team Look Like?

Dedicated Team Advantages

Regarding critical mass, if you have all the automation expertise in one place, the automation endeavor has less friction when it comes to helping one another conquer automation challenges and deliver new automation capabilities. New automation engineers can be brought up to speed quickly because they have the rest of the automation team “at arm’s reach”. The walls between who automates what don’t need to exist (except when funding comes into play, but, to quote Alton Brown, that’s another show).

Following from, and dependent on, critical mass is the ability to gain an economy of scale. The Economy of scale, for our purposes, is being more productive, efficient, effective, and therefore economical in the delivery of automation. Speaking practically, this means that there is a consistency that can be attained by having a dedicated automation team and that consistency can scale because of the critical mass and consistency. The dedicated team can be indoctrinated with the organization standards and the organization can preside over the guidelines; automation, regardless of framework or library, can be implemented based on a consistent ideology. When implemented judiciously, this approach can deliver economic gains to the organization and the company.

Dedicated Team Limitations

What Does A Distributed Team Look Like?

In the case of automation specialists, often called software development engineers in test (SDETS) or quality engineers (QE), the name describes the role. The automation specialists only do that: automation. In the microcosm, we have that economy of scale and specialization, but it’s a relatively small economy in the scope of the overall company because it’s exclusive to the team on which the automation specialists work; this is especially true if the company has many, similar delivery teams. In this team implementation, the automation specialists are not testers/QAs; they work with the testers/QAs as well as the developers to develop the automation that helps the team be more effective or more efficient at testing.

It’s a great story to say that the testers/QAs do the automation. They automate instead of “doing it manually” and the team “goes faster”. To accomplish this approach, the testers/QAs need to have time to gain the skills necessary for automation; they also need the time to create and maintain the automation. Often, the whole team does automation variant can be more effective. In this model, the automation work, like other work, is not assigned to one role; anyone on the team with the appropriate skill set is empowered to perform or help with any task. Developers know how to program so they can be a natural fit to create automation or pair with a tester to create automation.

A third, and recently hot, approach is “we don’t need testers” because we can have the developers do the automation. This has a direct correlation to the testers/QAs do the automation approach, namely that automation is something you “just do” without training or special skills; since automation is programming and developers know how to program, it’s a natural fit for them. Like testers, however, developers need time to gain the appropriate skills to appropriately implement automation.

Advantages

Additionally, the deep domain knowledge enables the automators to discover hence unthought-of paths in the code that merit testing. This is difficult-to-impossible without the deep domain knowledge this approach permits.

Distributed automation teams are especially appropriate when the teams are working disparate products where there is little-to-no overlap between the application or system technologies.

Limitations

That’s OK, let’s “just” have the testers/QAs do automation. Bzzt, play again. There is an unfortunate perception that automation is something that you “just do”. I just brush my teeth, I just eat lunch, I just run some test cases, and I just create some automation. As much as it would be great if this were true, it’s just not. What typically happens is that the testers/QAs are “tasked” with automation with little-to-no guidance or training and this effort is heaped on top of their existing work, which continues to grow; when was the last time a team released a new version of their software with fewer features? Unless the organization is tolerant of a “let’s slow down now to go faster later” strategy, this approach is fraught with peril and the likely outcome will be “automation can’t work here”.

Developers do the automation has a similar issue. Leadership wants this to happen with negligible impact to effort estimates even in the short term. For teams that can’t embrace the “let’s slow down now to go faster later” strategy, one of two outcomes is usually the result. One outcome is that the created automation is largely confirmatory; the automation shows that the software can work under very specific conditions and other conditions are not accounted for; the developers are often required to perform heroic efforts to accomplish even this due to needing to attain an on-time delivery. The other outcome is that automation takes “too long” so it doesn’t work here, and the automation endeavor is abandoned.

If the automation creation is truly distributed, it is difficult or impossible for the teams to leverage each other’s work. Even the most well-meaning teams that try to develop cooperatively will have trouble doing so in a consistent manner. The incentive for the individual teams is to achieve the goals set for those teams. This work is not purposely completed to the detriment of the other team, but there is usually little-to-no incentive for teams to cooperate at that level. Senior leadership is usually not thrilled to pin their organization’s success on the “goodwill” of another team; similarly, those same leaders are not likely to jeopardize their organization’s success by spending some of their team’s cycles to help another team’s automation implementation if the need comes at a critical time.

If code sharing is still desired, some of the hard dependencies can be mitigated by having a shared codebase, something like an “internal open source” repository. The primary challenge there is when conflicting changes are made to the shared code. How is that conflict resolved? Often, even in the best-intentioned companies, organizational incentives and directives reign so the teams agree to fork the shared automation repository with the well-intended agreement to merge after the next business milestone. Sadly, that seldom happens; there’s always another business milestone.

The Hybrid Approach — An Oft-Appropriate Middle Ground

As noted above, there are advantages and disadvantages to each of the preceding approaches. What if we meld the two? Let’s take the strengths of each team layout and attempt to mitigate the disadvantages. The goal is to let each of the disciplines focus on its strengths.

In this approach, the automation development is divided into two competencies: the base team and the distributed teams.

The Base Team

The base team comprises experienced automation developers who are in the best position to provide a general framework, platform, and infrastructure for automation creation. The base team is also tasked with the training, coaching, and guidance of the teams implementing the actual automation.

It’s also responsible for coaching on the appropriate practices for the company. This team is trained in automation strategy, implementation, and theory. They provide the base infrastructure implementation and many, if not all, of the shared components used across the distributed teams but they understand and can coach on the theory behind the strategy and implementation.

The Distributed Teams

Risks

If the base team isn’t staffed with enough appropriate people to service all of the distributed teams, the base team will become a bottleneck, actually impeding progress instead of facilitating it. If getting enough appropriate base team staff members is a challenge, the previously mentioned internal open-source model can be valuable. This model has a higher chance of success with a base team than with a purely distributed model because the base team is the steward of the shared code. Care must be taken, however, to avoid the next risk.

A dedicated base team runs the risk of becoming or at least being viewed as an ivory tower, i.e., delivering intellectual solutions and directives as opposed to practical and workable ones. One way to mitigate this risk is to give the base team a directive to be collaborative as opposed to dictatorial, conversations as opposed to edicts. Another way is to include members of the distributed team as stakeholders, which they indeed are, in any prioritization and feature creation discussions.

Sadly, as in many other corporate aspects, politics often play a role when organizing a base team. The base team is seen as strategic: whoever controls the base team owns their own automation destiny. When one application team or organization controls the base team, the non-controlling teams and organizations are at a disadvantage when conflicts arise or when the controlling team must make difficult business decisions. One way to mitigate this situation is to organize the base team in a “shared services” organization, such as the organization that houses IT and security.

Recommendations?

[Editor’s Note: We are grateful to Paul Grizzaffi for contributing his blog post. If you are looking for some real-world examples that echo Paul’s hypotheses and conclusions, check out Marie Drake’s webinar on testing a design system at News UK and Priyanka Halder’s webinar on high performance testing at GoodRX.]

For More Information

About the author

Originally published at https://applitools.com on August 11, 2020.

Deliver visually perfect web and mobile apps with AI-powered end-to-end visual testing & monitoring. Get started, free: http://bit.ly/ApplitoolsFree