Home▸Free interview questions▸AI system design interview questions
free · no sign-up · no loginAI system design interview questions, answered
20 questions with short answers, picked from the Q&A in the AI Engineer course. Tap a question to see the answer.
What is AI system design?
AI system design is planning the whole application around a model: where the data comes from, how retrieval works, which model answers, which tools it may call, the guardrails, evaluation, monitoring, cost and speed. Interviewers want a clear request flow and honest trade-offs, not a list of product names.
- Clarify the goal, users and limits
- Draw the request flow
- Pick models, retrieval and tools
- Cover safety, evaluation, cost, latency
No question matches that search. Try a shorter word.
AI system design interview questions
The full set, with labs and mock interviews, is in the AI Engineer course.
AI System Design
1What does a system design interview test?
Whether you ask the right questions, make sensible trade-offs, know how the parts fit together, and think about real problems like permissions, cost and failures.
2What should you do first?
Ask questions first: who the users are, what documents, how fresh they must be, who can see what, how fast it must be, and how success is measured.
3Why estimate scale out loud?
Numbers guide the design. For example, 200,000 documents of 5,000 tokens is about 1 billion tokens. That tells you the cost, storage and how big the system must be.
4How do you design for latency targets, like the first word within one second?
Give each step a time budget, stream the answer, keep prompts short, cache, use smaller models for early steps, and run steps at the same time when possible.
5Describe the high-level architecture of a RAG assistant.
Two parts. Ingestion: collect documents, read them (with OCR for scans), chunk, add details, embed, and store. Query: rewrite the question, filter by access, search, re-rank, and answer with sources.
6How do you keep the index fresh?
Update only documents that changed, remove deleted ones, and run a full check now and then.
7How do you control cost at scale?
Cache common answers per access group, use small models for rewriting questions, reuse shared instructions, and set cost and time limits for each step.
8How do you handle permissions properly?
Save access groups on every chunk and filter by the user's groups, from their login, before searching. Restricted text never reaches the model.
9What failures do you plan for?
Plan a backup for each part: if the vector database is down, use keyword search; if the model provider is down, use a backup model; treat document text as data; and date every document.
10What trade-offs do you name at the end?
Chunk size versus precision, re-ranker quality versus speed, hosted versus self-hosted control, and how you will measure success.
11Design a customer support agent that can issue refunds.
Classify each message, answer policy questions from documents with RAG, and give the agent a refund tool with rules enforced on the server: check the order, the amount limit and the policy. Refunds above the limit go to a human. Log everything and test on past tickets.
AI System Design: The Building Blocks
12What does an API gateway do in front of an AI system?
It is the single front door. It checks who is calling, applies rate limits, routes the request to the right service and records it. Without it, every service would repeat the same checks.
13When do you put a human in the loop?
When a mistake is costly or cannot be undone, such as a refund, a payment or a message to a customer. The AI prepares the action and explains why. A person approves, edits or rejects it. Low-risk actions can run on their own.
14Where do you check identity and permissions, and why not in the prompt?
At the gateway and again in the service that owns the data. A prompt is only a request to the model, and a clever message can talk the model out of it. Code that checks the user and filters the data cannot be talked out of anything.
15Where do guardrails sit, and what do they check?
On both sides of the model. Input guardrails mask personal data and block attacks and off-topic requests. Output guardrails check policy, check the answer against the sources and check the format. Risky cases go to a human.
16How does a semantic cache work, and when is it unsafe?
It saves an answer with the meaning of its question. When a new question means the same thing, the saved answer is returned with no model call. It is unsafe for personal answers and for facts that change, so cache by user or tenant and set an expiry.
Solution Architecture, System Design and Technical Scoping
17Give one non-functional requirement.
For example: answer in under 3 seconds, or 99.9% uptime.
18Where do you check who the user is?
At the edge, before anything else runs.
19The main model is down. What should happen?
Fall back to another model or a safe message.
20What goes into an architecture decision record, and why write one?
Four short parts: the context, the options considered, the decision, and what you gave up. It takes ten minutes to write. Months later it tells the customer's team why the system is built this way, and stops the same debate from starting again.
More free interview questions by topic
All 91 free questionsAgentic AIRAGMCPLLMLLM evaluation and guardrailsFDE
Now test yourself
Play the free quiz, or try a mock interview where you answer out loud and get a score.