For years, system design interviews meant “Design YouTube,” “Design Uber,” or “Design WhatsApp.” Corporations hiring AI Engineers, Utilized Scientists, and GenAI Engineers now ask a special set of questions: “Design ChatGPT,” “Design a buyer assist AI,” “Design GitHub Copilot,” “Design an AI code reviewer,” “Design a authorized doc assistant.”
Most engineers can name an LLM API. Fewer can clarify the encompassing structure and defend the design decisions below strain. That second ability is what these rounds check. We will provide you with one reusable framework that works throughout all of those prompts, so that you cease memorizing a separate reply for every.
# Why The Questions Modified
AI hiring grew quick sufficient to reshape the interview course of. AI Engineer was ranked the #1 fastest-growing job within the US for the second 12 months working, with postings up 143% 12 months over 12 months in 2025.
LinkedIn knowledge reveals that the position added 75,000 US postings between 2023 and 2025, and that the share of AI and machine studying jobs within the tech market rose from 10% to 50% over the identical interval.
With that quantity, the questions shifted towards AI-first software program. IGotAnOffer reviews that these rounds now heart on the way you wrap giant language fashions (LLMs) into merchandise: designing agentic loops, integrating retrieval, and reasoning about price. Deep data of mannequin internals issues lower than it used to.
A practitioner discipline information constructed from late-2025 and early-2026 interview reviews lists the commonest prompts as designing an AI chatbot, a doc question-and-answer or retrieval-augmented technology (RAG) system, an AI coding agent, and a voice assistant.
# What Interviewers Consider
Educative describes the shift nicely: these interviews check your skill to cause about probabilistic, cost-constrained methods reasonably than deterministic CRUD providers. The core ability is navigating trade-offs amongst latency, price, high quality, and security when these pressures pull in reverse instructions.
Sturdy candidates clarify why every layer exists and what breaks with out it. Naming the layers with out that reasoning reads as shallow.
Senior-level reviews say interviewers choose 3 to five areas and drill into failure modes and “what went mistaken final time” reasonably than skimming many subjects. What units folks aside is manufacturing expertise and a willingness to speak about what they’ve really shipped.
# The Framework
Throughout guides, the identical sequence recurs. The System Design Handbook lays it out as eight steps; we use seven.
- Make clear. Pin down the information sources, privateness guidelines, latency price range, tolerance for factual errors, anticipated scale, freshness wants, and whether or not you may name a third-party API or should self-host.
- Estimate. Work out tokens per second, context window measurement, embedding quantity, price per name, and peak queries per second (QPS).
- Sketch the structure. A defensible default flows by way of an enter layer, a security and personally identifiable data (PII) layer, an orchestrator, retrieval (a vector database plus a reranker), the mannequin (routed by activity problem), post-LLM guardrails, response streaming, and observability.
- Deep dive. Decide one or two parts and go deep: RAG technique (chunking, hybrid BM25 plus dense retrieval, reranking), immediate design, caching (actual and semantic), and mannequin tiering.
- Commerce-offs. Say them out loud: latency versus high quality, RAG versus fine-tuning, price ceilings, fallback fashions when capability is tight.
- Failure modes and observability. Hallucinations, immediate injection, supplier outages, embedding drift, multi-tenant isolation, and the way you’d detect every.
- Evolution. A/B immediate testing, suggestions loops, eval gates earlier than launch, and gradual mannequin migration.
The only most reported failure is leaping to an answer earlier than clarifying necessities, constraints, and success standards. Spend the primary couple of minutes on step 1.
# The Primitives You Want To Know
Most “Design X” prompts reuse the identical components. Know these 5 nicely sufficient to attract and defend them.
// Retrieval-Augmented Technology (RAG)
At its core, a RAG system has a question encoder, a retriever that fetches a ranked listing of paperwork from a corpus, and a generator that circumstances on each the question and the retrieved context.
Manufacturing deployments add doc chunking, embedding pipelines, vector retrieval, caching, and analysis logging, and so they implement entry boundaries so customers can’t pull knowledge they need to not see.
RAG alone typically cuts hallucinations by roughly 40 to 71%.
// Mannequin Routing
Value and latency are actual constraints, so say the way you deal with them.
GPT-4-tier fashions price about $10 and $30 per million enter and output tokens, reply in 3 to five seconds, and an agent dealing with 10,000 conversations a day at 5,000 tokens every runs previous $7,500 a month on a single supplier.
Ship routine requests to low-cost fashions and reserve frontier fashions for the laborious ones. Since 60 to 80% of agent requests are routine, routing normally saves 40 to 70%.
Routing, semantic caching, immediate compression, and streaming collectively lower prices by 40 to 60% whereas holding high quality regular.
// Guardrails
Guardrails sit at two layers.
Pre-LLM handles enter validation, PII redaction, and prompt-injection protection.
Submit-LLM handles schema enforcement, refusal insurance policies, and fact-checking in opposition to the retrieved context.
Layered guardrails — system prompts, RAG grounding, quotation enforcement, confidence scoring, and monitoring — can scale back hallucination threat by 71 to 89% in opposition to a baseline price of three to twenty%. Current these as ranges within the interview, for the reason that figures come from combined sources.
// Analysis And Observability
Log mannequin variations, retrieval metadata, software traces, security selections, latency, and price per request, utilizing immediate hashes reasonably than uncooked textual content.
Mix offline evals (LLM-as-judge calibrated in opposition to floor reality) with on-line metrics reminiscent of faithfulness, context recall, and reply relevance. The identical logs assist debugging, security, and compliance directly.
// Agentic Loops
For agent-heavy designs (an AI code reviewer, a analysis assistant, a buyer assist agent), the sample is request consumption, context meeting, LLM reasoning, motion validation, sandboxed execution, consequence processing, state replace, then loop or cease.
Maintain issues separate: the LLM causes, the orchestrator controls stream, the coverage engine governs, and the sandbox executes.
# Reference Architectures Price Naming
Citing an actual system reveals you’ve got learn previous the tutorials.
GitHub Copilot is nicely documented. Its built-in growth setting (IDE) extension extracts the code earlier than and after the cursor, together with contextual indicators reminiscent of open information, imports, and language metadata, to assemble a immediate for the underlying mannequin.
Fill-in-the-Center (FIM) gathers neighboring tabs and file-path headers and sends the assembled immediate to GitHub’s backend, which filters for security and routes it to the mannequin working on Azure. FIM offers roughly a 10% relative elevate in acceptance over prefix-only prompting.
GitHub additionally runs a separate mannequin to attain completions on high quality and security, refined by way of offline, pre-production, and manufacturing evals.
Different methods price a sentence:
- Uber’s GenAI Gateway with a PII redactor throughout 60+ use instances.
- Airbnb’s conversational AI with chain-of-thought reasoning and guardrails.
- Perplexity serving 200M each day queries on Vespa.ai.
- Slack’s stateless RAG with fashions in an escrow digital non-public cloud (VPC).
- Anthropic’s multi-agent analysis system with an Opus orchestrator and Sonnet subagents.
# Widespread Errors
Just a few errors present up many times, and most are straightforward to repair when you look ahead to them.
The most important is designing earlier than clarifying. Candidates hear “Design ChatGPT” and begin drawing containers inside seconds, skipping the necessities, constraints, and success standards that ought to form all subsequent decisions. That is the highest reported failure in interview debriefs, so spend the primary couple of minutes asking questions.
A second is itemizing parts with out saying why every is there. A diagram with a vector database, a reranker, and a guardrail layer means little in case you can’t clarify what breaks once you take away every bit. Interviewers learn that as memorization with out actual understanding.
A 3rd is skipping price and latency. These are first-class constraints in AI methods. A design that ignores the token invoice or the 3- to 5-second response time is incomplete, nevertheless clear it seems to be.
The final is forgetting failure modes. Sturdy solutions cowl hallucinations, immediate injection, supplier outages, and multi-tenant isolation, plus how you’d detect every. And once you quote hallucination-reduction numbers, give ranges, for the reason that printed outcomes fluctuate by supply and setup.
# Conclusion
AI system design interviews reward a repeatable course of greater than a memorized reply. We checked out why the questions modified: AI hiring grew quick, and firms now ask you to design merchandise constructed round LLMs reasonably than basic machine studying pipelines.
We checked out what interviewers consider: clear reasoning about probabilistic, cost-constrained methods, and trustworthy speak about trade-offs and what has gone mistaken in manufacturing.
The core is the 7-step framework. Make clear the issue, estimate the load, sketch the structure, dig into one or two parts, identify your trade-offs, plan for failure modes and observability, and say how the system evolves.
Underneath that sit 5 primitives it is best to be capable to draw and defend: RAG, mannequin routing, guardrails, analysis, and agentic loops. Realizing one or two actual architectures, reminiscent of GitHub Copilot, offers your reply weight.
Observe the framework on a couple of prompts, and “Design ChatGPT,” “Design an AI code reviewer,” and “Design GitHub Copilot” begin to really feel like the identical downside with completely different inputs.
Nate Rosidi is an information scientist and in product technique. He is additionally an adjunct professor instructing analytics, and is the founding father of StrataScratch, a platform serving to knowledge scientists put together for his or her interviews with actual interview questions from high firms. Nate writes on the newest tendencies within the profession market, offers interview recommendation, shares knowledge science tasks, and covers all the pieces SQL.
