ML system design interview: a framework that works
A six-step framework for the ML system design interview: requirements, metrics, data, model, serving and monitoring, with a worked example and mistakes.
On this page
An ML system design interview asks you to design a machine learning system from a one-line prompt, such as "design a product recommendation system", in 45 to 60 minutes. The framework that works is six steps taken in order: requirements, metrics, data, model, serving, and monitoring. Interviewers are not waiting for one correct architecture; they are scoring how you structure an open problem and how you reason about trade-offs.
What does an ML system design interview test?
An ML system design interview tests whether you can take a vague business problem and turn it into a system that would work with real data and real users. The interviewer is looking at four things: whether you clarify before you build, whether you understand that data matters more than model choice, whether you know how a model is served and kept healthy, and whether you can explain trade-offs clearly.
The round is common from mid-level upwards. Where the round sits among the others is covered in the overview of the GCC AI interview process.
What framework should you use for ML system design?
Use a fixed six-step framework and say it aloud at the start, so the interviewer knows where you are going. A fixed order stops the most common failure, which is jumping to a model in the first two minutes.
Clarify requirements
Who uses the system, what decision it makes, scale, latency, and what a wrong answer costs.
Define metrics
One business metric, one offline model metric, and guardrails such as latency or fairness.
Design the data
Sources, labels, features, freshness, and how you avoid leaking the future into training.
Choose the model
A simple baseline first, then a stronger model only where the baseline falls short.
Design serving
Batch or real time, feature lookup, latency budget, fallbacks, and rollout.
Monitor and iterate
Drift, data quality, feedback loops, retraining triggers, and online experiments.
Step 1: Clarify requirements
Requirements come first because the same prompt can describe very different systems. "Detect fraud" for card payments needs an answer in milliseconds; "detect fraud" for insurance claims can run overnight. Ask who the user is, what action follows the prediction, how many requests arrive, how fast the answer must be, and which error is worse: a false positive or a false negative. Then state your assumptions and move on.
Step 2: Define metrics
Metrics come second because they decide every later choice. Name the business metric the system should move, then an offline metric you can compute before launch that tracks the business metric reasonably well. For a ranking problem that might be a ranking quality measure at the top few positions; for rare-event classification it is usually precision and recall at a chosen threshold, not accuracy. Add guardrails: latency, cost per request, and any group the system must not treat unfairly.
Step 3: Design the data
Data deserves the most time because it limits everything the model can learn. Say where the labels come from and how trustworthy they are. List feature groups rather than fifty individual features: user history, item attributes, context such as time and device. Then cover two risks. Leakage means a feature contains information that would not exist at prediction time. Training-serving skew means a feature is computed one way in training and another way in production. Split training and test data by time when the problem has a time order.
Step 4: Choose the model
Start with a simple baseline because a baseline tells you how hard the problem is and gives you something to ship. A heuristic or a linear model is a fair first answer. Then propose a stronger option, such as gradient-boosted trees for tabular data or a two-stage retrieve-then-rank design for large catalogues, and say what evidence would justify the added complexity. If you need to revise the building blocks, start with machine learning fundamentals.
Step 5: Design serving
Serving design decides whether the model is usable. Choose between batch prediction, where scores are computed ahead of time and looked up, and real-time prediction, where the model runs on each request. Batch is cheaper and simpler; real time is needed when the input is only known at request time. Mention where features are fetched from, the latency budget, a fallback when the model or a feature is unavailable, and a gradual rollout so a bad model reaches a small share of traffic first.
Step 6: Monitor and iterate
Monitoring closes the design because models decay after launch. Cover input drift (the data changes), prediction drift (the score distribution changes), and delayed labels (you only learn the true outcome later). Say what triggers retraining and how a new model is compared with the old one in an online experiment. The practices behind this step are explained in MLOps basics.
How should you split your time in the interview?
Split a 45-minute round so that no step is skipped: roughly a fifth on requirements and metrics, the middle on data and model, and the last quarter on serving and monitoring.
| Step | Share of time | What the interviewer hears |
|---|---|---|
| Requirements and metrics | First 5 to 8 minutes | You do not build before you understand |
| Data and features | About 10 minutes | You know where model quality comes from |
| Model | About 10 minutes | You can justify complexity |
| Serving | About 8 minutes | You have shipped, or can reason like someone who has |
| Monitoring and iteration | Last 5 to 7 minutes | You think beyond launch day |
What does a strong answer sound like?
A strong answer sounds like a series of decisions with reasons. Take the prompt "design a system that recommends products on a shopping app's home page". A strong candidate confirms the goal (more purchases, not just clicks), the scale (a large catalogue, many users) and the latency limit (the page must load quickly). The candidate then proposes two stages: a cheap retrieval step that narrows the catalogue to a few hundred candidates, and a ranking model that orders them using user and item features. New users with no history get popular items in their region as a fallback. The candidate ends with an online experiment that compares purchase rate against the current page.
Weak answer
- Names a deep learning model in the first minute
- Never asks about scale, latency or the cost of errors
- Uses accuracy as the only metric
- Lists tools instead of explaining choices
- Stops at training; no serving or monitoring
Strong answer
- Asks four or five questions, then states assumptions
- Ties an offline metric to the business goal
- Starts from a baseline and earns each added component
- Names trade-offs: freshness against cost, recall against latency
- Ends with drift, retraining and an online experiment
What are the most common ML system design mistakes?
The most common ML system design mistake is treating the round as a modelling question. The others follow from the same habit.
- Skipping requirements. A design without a latency limit or an error cost cannot be evaluated, so the interviewer cannot give you credit for it.
- Ignoring where labels come from. Saying "we train on labelled data" without saying who labels it, and how late labels arrive, misses the hardest part of many real systems.
- No fallback. Production systems fail. A default answer when the model is down shows operating experience.
Is LLM system design different from ML system design?
LLM system design uses the same six steps, with different content in the data, model and monitoring steps. The data step becomes document ingestion, chunking and retrieval; the model step becomes a choice among prompting, retrieval-augmented generation and fine-tuning; and monitoring becomes evaluation of groundedness, answer quality, cost and safety.
For a prompt such as "design an assistant that answers employee questions from internal policies", cover access control on documents, how retrieval quality is measured separately from answer quality, and what the assistant does when the documents do not contain the answer. The mechanics are covered in how RAG works and LLM evaluation basics, and likely follow-up questions appear in GenAI interview questions.
How do you practise ML system design?
Practise ML system design aloud, with a timer, on four to six different problem shapes: reading designs builds recognition, but only speaking builds the ability to produce one under time pressure. The design round rewards real project experience, so a project you have built end to end is the best preparation of all. The free readiness check on The GCC Engineer shows which design-related skills your target role expects and where to start.
Key takeaways
- Spend the first five to eight minutes on requirements and metrics before proposing any model.
- Start with a simple baseline and explain what evidence would justify something more complex.
- Data and labels deserve more time than model architecture in most designs.
- Always close with monitoring: what can go wrong after launch and how you would notice.
- State each trade-off aloud; the reasoning is what gets scored.
Frequently asked questions
Is ML system design asked for freshers?
A full ML system design round is uncommon at entry level. Freshers are more likely to get a lighter version inside a fundamentals round, such as 'how would you approach building a spam classifier'. The same six steps still help: talk about the goal, the data, a baseline, how you would evaluate it and what might go wrong.
How is ML system design different from a normal system design interview?
A software system design round centres on components, storage, scaling and consistency. An ML system design round adds the parts that only exist when a model is involved: labels, features, training and evaluation, the gap between offline and online behaviour, and model monitoring. Backend-leaning AI roles can get either type, so ask the recruiter which one to expect.
Do you need to draw diagrams in an ML system design interview?
Yes, a simple one helps. Draw a box-and-arrow diagram with the data sources, the training pipeline, the model store, the serving path and the monitoring loop. Keep it rough. The diagram gives you and the interviewer something to point at, and it stops you forgetting the serving and feedback paths.
How many practice problems are enough for ML system design?
Four to six problems practised properly are worth more than twenty read passively. Choose different shapes: a ranking or recommendation problem, a classification problem with rare positives such as fraud, a forecasting problem, and an LLM application such as document question answering. Practise each aloud with a timer, then compare against a written design.
What if the interviewer keeps changing the requirements?
Changing requirements is deliberate. The interviewer wants to see whether your design adapts. Say which parts of the design the change affects and which stay the same, then adjust only those parts. For example, a tighter latency limit changes the model size and serving path, but usually not the labels or the offline metric.