Building a Winning DevOps Team

DevOps, and Ops in general, is one of the most important components of a company's production machine. Investing in building a winning DevOps team should be a priority for every company and will pay off quickly.

As organizations and operations grow (products, features, number of customers, etc.), the scale of traffic, data, OS types and so on, increase rapidly. More products to support, new features to release and a lot more in-house cycles (e.g., QA, CI\CD, SME, SRE) need to be managed. A DevOps team's first mission is to build, maintain and support the company’s technology stack. Whether it’s a PaaS, SaaS or IaaS environment, microservices or monolithic, docker or K8S, it does not matter--DevOps engineers need to be able to support any environment the company chooses at any time and anywhere (On Premises, Data Centers, Cloud). Here are the subject areas any organization and it’s DevOps engineers have to deal with:The list goes on, and every item is its own area of knowledge. It's “a small world” of expertise, and any DevOps engineer is expected to master it all (or at least a large subset of it ¯\_(ツ)_/¯). For example, let's say your company is building the next edge, top of the line API serving tool. The tool is web based and expected to serve TB of daily transactions. TB of events pounding your chosen big data tools, logging everything for sanity, monitoring everything so you’ll be able to wake anyone up when necessary, profiling, ingesting, segmenting…It’s growing and doing so exponentially. Recruiting and finding DevOps engineers to support this environment is challenging. Finding someone who understands all the aforementioned subjects is rare, and if you do find that person, they'll cost you accordingly. (Let's assume money is not an obstacle right now and focus on finding the right engineer to fulfill, create and maintain your environment.)

A Six Step Process

My strategy is divide and conquer. Let's break down and define what we're looking for. Defining the interview process is crucial for successful hiring. A good process will result in your finding a quality candidate (technical, personal and cultural) and will help maximize the candidate’s compatibility with the company and role. Throughout my years of experience, I’ve defined a 6 step process that helps me form and create winning engineering teams:

• Step 1 - Define the role, and sync with the company recruiter.
• Step 2 - Phone screen.
• Step 3 - Break down the skills to interview modules.
• Step 4 - Prepare questionnaire for each module.
• Step 5 - Form your interviewer squad and begin interviewing.
• Step 6 - Closer. All interviewers meet and decide.

Step 1. Define the role, and sync with the company recruiter.

The company recruiter will be your liaison to the candidate and will have the first interaction with them. It’s important to follow these steps:

• The recruiting manager must have a conversation with the recruiter and make sure they understand the job description and the role's technical requirements. Talk with them, explain yourself, explain the role and expectations. Don't skip this stage. If you do, you'll end up wasting everyone's time in the long run.
• The recruiter should have a phone conversation with the candidate. The purpose of this conversation is to understand if the candidate is a good fit for the company and how the position fits within the company culture, the candidate's career development, and so on. This call also involves verifying the content of the candidate's CV.

The recruiter should pass their feedback to the hiring manager and if both gives thumbs up then proceed to Step 2.

Step 2. Conduct a phone screen interview.

Phone screen interviews are very helpful and significant time savers. Since the process is time consuming and involves numerous engineers interviewing the candidate, it’s important your time is well spent. You should respect the candidate’s time as well. After posting\advertising the job, people will start applying. Your time is precious so treat it accordingly! Even if the CV is appealing and it seems like the candidate is a perfect match, you still have to set up two phone calls, each being no more than 15-20 minutes:

• Recruiting Manager Interview - The recruiting manager needs to talk with the candidate to understand if the person is qualified and is a good fit for the team and company culture. They should also use the time to find out whether the candidate thinks the role meets their career aspirations, as well as validate the candidate's CV.
• Technical Interviewer - This can be any existing DevOps engineer or software engineer who can ask a series of easy, medium or hard questions related to the posted position and candidate's background.

The recruiting manager and the technical interviewer should meet following these interviews.  If both give thumbs up, proceed to step 3.

Step 3. Break down the skills to interview modules.

Interview modules represent a subject to discuss during the interview. For example:

• System + Networking
• Methodologies\Processes - Agile, CI\CD, etc.
• Software Engineering, Architecture Design
• Problem solving, debugging, monitoring
• Culture fit
• Practical\Code challenge

Step 4. Prepare a questionnaire for each module.

For each module, prepare a questionnaire that applies to different skill levels (junior, intermediate, senior and expert). This is best organized in a question and answer format as the answer section will be helpful in situations where the answer differs from what is expected.System + Networking module example:These questions will assess a candidate's skills and knowledge in areas of your specific flavor of operating systems and scalable networking. It is very important that these areas are well understood to allow for effective troubleshooting and competency in a technical environment. Lacking this basic knowledge can result in a long learning curve.

• How does traceroute command work?
• What is ARP?
• What is jitter/latency/packet-loss-ratio?
• What is proxy ARP?
• How do you set up a TCP connection?
• Etc.,

System engineering module example:The intent of this module is to determine whether the candidate has a reliable and repeatable process for solving complex problems. The interviewer will present a subset of the problems outlined below and progressively increase the complexity of the required solution through discussion with the candidate. If the problem field of knowledge is unknown to the candidate, only the process used to arrive at a solution should be judged and not the solution itself:

You are to architect a new email system capable of supporting an expanding user base with low delivery times. The incoming mail throughput is expected to be around a million emails per minute.

• Describe in detail or draw the mail system you would implement.
• What software choices would you make for each layer and why?
• What are your kernel, networking, storage and monitoring considerations?

Step 5. Form your interviewer squad and begin interviewing.

Each interviewer should pick a module that fits with their skill set so that they're able to ask and understand the candidate answers. Every interviewer takes notes and gathers feedback. Best of luck for a great interview!

Step 6. Closer. All interviewers meet and decide.   

All interviewers meet and share thoughts and concerns. At the end of the meeting, ideally each person will be able to give a thumbs up or down. If, however, there's a lack of clarity or unresolved concerns about the candidate, you can set up an additional interview before making a final decision. I've been using this method for a couple of years and it's proven to be very successful. Good Luck!Adi Glasman is a Senior Operations Manager at APAC - Zendesk. You can find more articles by Adi, as well as the original post here.