Rock Solid Software Without Hiring an Army

This week we’re attending the Business of Software conference in Boston, which always provides a plethora of inspiration. We wanted to share some of the highlights in our notes… especially this one, which is so relevant to all of us who care deeply about developing quality software. Please pardon the hastiness… we're still in sessions...So chances are you’re already selling ahead of your roadmap and your dev team is getting pretty big. In this talk, Trish Khoo, Engineering Manager at Google, outlines approaches to keeping pace and maintaining high quality without hiring an army, drawing on a decade of software testing at Campaign Monitor, Google and Microsoft.

---

We all know about the iron triangle of software development – the tradeoff between Quality/Nice things, Time (aka $$), and People (also $$$). This triangle is what we use to justify why we can’t have nice things (ahem...).At Google, they try to live in a triangle-free world and have it all – at speed –without hiring a huge army of people. It is possible. Google has some pretty hefty reliability requirements, so quality is essential:

• 1B active monthly users of Google Maps services
• 1B downloads of Google maps on Android
• 2M active websites and apps use the Google Maps API

  Q: At your company, is there a problem coming your way? “We often find that when doing due diligence, that the team got the thing up and running as fast as possible and that’s fine for getting to £1-5 pounds revenue but they’ve created scaling issues for themselves after that.” - Simon Menashy, MMC Ventures   So what do you do?Typical decision #1 – Hiring a QA team to fix the quality problemDon’t hire the QA army. QA is accountable but has very little power to change anything. Manual testing is very slow.

How many QA people does it take to change a light bulb?

None. They will just tell you the room is dark.

Manual testing is the best way to test a simple product but it doesn’t work for complex ones.  Typical Decision #2 – Hiring an automation team to fix the testing problem Write automated tests to replace the manual ones. A bad sign:

Subject – we’re hiring!

Test Automation Engineer

Low coding skill is ok

Write all our tests for us

We have a foosball table and a blimp

 In other words - a very low bar. Another bad side: you teach your manual testers how to write tests. You end up with big unmaintainable piece of software to test your software. Which basically amounts to hiring a team of people to encourage bad practices in your Dev team. And then there are the Triage meetings: what is the least crappy product we can live with in production in order to get the thing out the door? ** Seven years ago, Google’s teams released to production once or twice a month; with the majority of time taken by testing. Today Google teams release to production daily or sub-daily. There was no decrease in quality.The secret: The whole team must commit to quality.It’s not a separate group where quality is your job but you can’t do anything about it. What testing really is: Testing provides expert driven feedback on the state of business requirement gaps, user impact and overall project quality. One of the ways we do this is by checking things. Everybody can check things. It’s easy. Just look!Checking is easy. Testing is hard.A Testing expert in your dev team will…

• Make sure people get info about your product at the optimal time to make timely decisions.
• Make your dev process more efficient
• Design tools and infrastructure to make your devs more productive

So what can you do to build rock solid software without hiring an army? 4 key things…  1: Change your company culture…It has to come from top down if it’s going to change.The Pushback…

• We don’t have time to write tests
• We don’t know how
• That’s not the way we work at this company

And you really need a predictable level of “done”. Management needs visibility – otherwise they think it’s taking longer to get to the right level of “done.” But really we’re reducing the big testing phase at the end and getting everything done faster.2: Hire the Right SkillsEngineering Productivity – you need these 2 key roles:

• Software Engineer, Tools & Infrastructure
• Test Engineer – this role is higher in test experience. Also an engineer. They look for ways to make testing more efficient, look for new tools and ways to make things better. (At Google, the technical bar is not quite as high technically as for Software Engineers – but it’s balanced by their testing experience)

Google hires these engineers at the same skill level as their other engineers – very high bar.Empower your testing expertThese people are hard to find, hard to retain!

• You have to make them a first class citizen within the team – their job is often to convince people to do things they don’t want to do
• Similar job title and salary to developers
• This role needs management support

3: Learn, don’t blame.

• Cultivate a postmortem culture
• Trust your people

Learn from your mistakes and get feedback. People learn it’s ok to make mistakes and take risks. If people feel like they’re going to be punished for mistakes, they’re not going to tell you about mistakes they or their coworkers have made. If it’s ok to make mistakes, you’ll learn about problems earlier than later… when they’re easier to fix.4: Launch and iterate.Faster releases mean faster feedback. So launch fast and keep improving.Do experiments and “dogfood”You want to be able to try things out quickly and on a small percentage of users to see how it works. It's also good to ‘dogfood’ (use your own products).  What’s in it for you?

• rock solid software
• ability to quickly change product direction
• freedom to innovate

You really can build high-quality software without hiring an army. But the whole team has to focus on quality from the beginning.  XebiaLabs develops enterprise-scale Continuous Delivery and DevOps software, providing companies with the visibility, automation and control to deliver software faster and with less risk. Learn how…