My strongest skill as an engineer is in distributed systems architecture. It's what I'm passionate about, it's what gets me excited to go to work everyday (if you want to see crazy-large distributed systems in production, Uber's the place!), and I think it's where the most interesting and challenging problems are found in the industry today. It's a big part of my job, and, consequently, I've given quite a few architecture interviews. In today's post, I want to share some of the things I've learned about how to structure and run these types of interviews. If you're here because you want to know how to ace the architecture interview, you should keep reading, because I think the best ways to prepare for these things is know how interviewers think about them.
First, why architecture?
Let's talk a little bit about why knowing something about the architecture of distributed systems is important. Chances are, unless you work at an extremely small startup, anything you work on runs on top of (and as part of) a large distributed system composed of many servers all communicating with each other over a network, and held together by a somewhat hacky yet sophisticated set of so-called "middleware" that lies between the bare servers and the top-level applications. If you're a software developer, you're going to have a difficult time writing really good code and building good applications if you don't know how the system works (at a very abstract and high level). If you're an infrastructure engineer, knowing the architecture in greater detail is essential. If you're in DevOps or SRE, there's no escaping the architecture. Basically, knowing the architecture of what goes on underneath the hood is critical for doing your job well.
Because of this, most companies have an architecture portion in their interviews, regardless of the specific role a candidate is being interview for. The big problem is that most people don't learn these skills in school, in bootcamps, etc., - more often than not, these skills have to be learned on the job, and there aren't very many resources online (or in textbooks) to learn from. This is partly due to the fact that distributed systems architecture has changed dramatically in the past fifteen years, it has been completely driven by industry, and academia hasn't quite caught up yet. So, how the heck do you learn all of this, and, when you have to give an architecture interview, how on earth do you do it?
If you don't have any experience in architecture yet, to learn what you need to know, you have to learn on the job. There's no way around it. Find out everything that you can about the architecture of the various systems you work on, and don't stop until you can stand in front of a whiteboard and architect each system off the top of your head. Ask experienced developers, ops engineers, and architects for reading materials (they can point you to good videos, talks, blog posts, and papers) and how they've architected their own systems. If your engineering organization runs architecture reviews, try to attend any that you can. Pay close attention when there are discussions about architecture in meeting or in talks. Once you get to know the terminology, you'll be able to learn more on your own. For example, if you hear someone talk about "pubsub messaging", spend some time googling it, ask more experienced devs, look through your own systems to see what kind of messaging (if any) they use, and why they do or don't use pubsub messaging (and also why they should or shouldn't!). If you have the opportunity to build a new application, service, or system from scratch, try to apply what you've learned, and have experienced engineers review your new application, service, or system. Eventually you'll get the hang of it - it's just like everything else in software engineering, the more you do it, the better you'll get.
Ok, now onto the interviews.
So let's imagine that you've been looped in to do the architecture portion of an interview. If this is your first time interviewing a potential candidate, it's nice to shadow another interviewer to see how they run things, and then (if possible) to have then reverse-shadow you (where you ask the questions and there's another experienced interviewer in the room to give you feedback after the interview). If you're at this point, you probably and hopefully have some solid engineering experience, and (most importantly) are very familiar with the architecture of your company's systems. You should know how each backend system works, how the whole thing fits together, why specific design decisions were made, what the tradeoffs of these design decisions are, any and all scalability limitations and potential points of failure, and what could have been done better.
Coming up with architecture questions shouldn't be very difficult if you have experience. You should be able to think of something you've worked on lately that makes a great question for a candidate to work through. I think that using questions based on your own experience make the best architecture interview questions - it allows you to talk about what you know, and it gives you a chance to see how candidates will approach real problems that they'll face on the job. Steer clear of the typical questions, like "how would you build and scale yelp?", "how would you build and scale gmail?", etc., unless you've actually worked (or will be working) on something similar. This point is worth repeating: ask about what you know, and what the candidate will be working on if they join your team. (Note: your company or team may already have a set of battle-tested, well-calibrated questions that you may need or want to stick to, which is fine, but make sure that you understand the questions fully and can work through them yourself and with others before asking them).
Let's assume that you have one hour to do the interview. Spend the first couple of minutes talking to the candidate, getting an idea of their past experience and where their strengths and interests lie. Then, pick from your list of architecture questions a system that the candidate will need to whiteboard from scratch. When you and the candidate are in front of the whiteboard, present the scenario ("design a system that does x, y, and z"). Don't give them a blank whiteboard, though. Blank whiteboards aren't helpful to candidates, and can make even the most experienced engineers nervous and cause them to freeze up - you'll have a hard time getting an accurate idea of how the candidate thinks through real-world problems if you don't give them anything to work with.
Present the system, talk a little bit about it, and then be the first person to draw out the basics of the system on the board. For example, if you're asking the candidate to build a system that checks the health of all hosts and applications, tell them what the hosts look like (and how many there are - start small!), what the applications look like (are there API endpoints they can hit for information? What kind of response will they get?), any problems they can expect to face, etc. Make sure to give them concrete numbers here (x number of servers, x number of applications, needs to meet x, y, and z targets in latency, availability, etc.), and tell them what you need and want from the system. Then ask them how they'd build the system, and let them begin.
Don't sit (or stand) back and watch like an all-knowing observer. Have a conversation, ask them what they're thinking, ask them why they're making certain decisions. Act like they are your coworker, and you're designing or reviewing a system together. This isn't a trivia game, it's not a chance to quiz someone on a specific technology or have them check off some magical list of "things-everyone-must-know-about-architecture" - it's a chance for you to see how the candidate approaches problems.
Don't expect them to know everything, or to know every technology. If they write something down, and you think it's a bad decision, ask them why they chose it, what its pros and cons are, what alternatives there may be. If you agree with their decisions, don't gloss over the details: ask them about their reasoning regardless here, too. Doing this back-and-forth can take a lot of time, so try to keep the pace moving briskly, and guide the conversation forward if it starts to lag. This isn't the time for deep discussions, just to get an understanding of their intuitions and gauge their knowledge about architecture.
You'll want to get to an initial solution by the end of the first half of the interview. They should have a simple solution for a small system after around 30 minutes, and should have explained their reasoning by this time - if it takes longer, the question may be too complicated, or you need to find new ways to make sure that the pace of the discussion is sped up a bit.
The next step is to scale the system. For example, instead of 50 servers, they now have 5,000 (or 50,000). Ask them how they would scale this, where they think things would break, and what changes they would make. Point out any points of failure that they don't notice, and ask what they would do to architect them away. If a SPOF can't be architected away, ask them how they would mitigate any problems that happened if (and when) it failed. This should take up the last half of the interview. By this time, you should have a very good idea of what the candidate knows, where their knowledge is lacking, how they approach problems, and (super important) how they solve problems when they don't know the answer. Make sure to leave a few minutes at the end of the interview for questions!
Before you leave the room, take a photo of the whiteboard. You'll need this when you write up your notes about the interview. Take some time - a few minutes - to write down any first impressions and any details about their architecture decisions that you may not remember. To make this easier, I always recommend writing short notes in relevant places on the whiteboard during the interview while you're talking through things (e.g., "async messaging for scalability", "SPOF", etc.).
I'm going to end this post with some thoughts on how to evaluate candidates based on their experience and the way they work through the architecture interview.
Junior candidates (especially those straight out of college or bootcamp) usually don't have any architecture experience, nor should they be expected to have any. I firmly believe that junior candidates shouldn't be interviewed on architecture at all. If you are asked to run an architecture interview for a junior candidate, make sure you remember how little you knew about architecture when you were a junior candidate, recognize that this will probably be a big learning experience for them, walk them through it very carefully, and design the question so that all of the relevant information is given to them (no guesswork!).
Candidates with over a year and less than four years of industry experience should be able to do moderately well on an architecture interview. They might not be able to scale the system very well (which is to be expected, because knowing how to do that requires a lot of hands-on, real-world, high-responsibility experience), but they should ace the first half. They might not know many different technologies, but they should know the basics of at least one relevant technology for each part of the stack (for example, they might not know the intricacies of every major database, but they should know one specific type of database very, very well). If a mid-level candidate can build the first simple system in the first half, and explain their decisions, the tradeoffs, etc., then they've done quite well.
Senior candidates should be able to ace both portions of the interview, building the first simple system quickly, and then scaling it up with ease. They should be able to easily discuss the pros and cons of each design decision, and should be able to see common problems and potential failure scenarios. With really experienced candidates, you should walk out of the interview feeling like you've learned something from them.
When interviewing senior candidates, be on the lookout for deep understanding of components of the system - especially if they come from a company where most of the infrastructure has been automated and abstracted away from developers. Some engineers (even with many years of experience) fall into the trap of never learning about the architecture of the overall system(s) they work with because they never had to. The easiest way to check for this is to ask them how something works - experienced candidates who don't know much about architecture will know all of the names and terms for things, but they won't be able to explain how these things actually work. If a candidate says something like "well, I never had to do that because we did that with automated-thing-at-previous-or-current-company," ask them how they would build automated-thing-at-previous-or-current-company from scratch.
That's how I think about architecture interviews, in a nutshell! I'd love if you'd comment with your thoughts and things you've learned about architecture interviews, both from the interviewer and candidate perspectives.