Home▸Free interview questions▸AI system design interview questions

free · no sign-up · no login

AI system design interview questions, answered

20 questions with short answers, picked from the Q&A in the AI Engineer course. Tap a question to see the answer.

20 questionsBy Samir KumarUpdated

What is AI system design?

AI system design is planning the whole application around a model: where the data comes from, how retrieval works, which model answers, which tools it may call, the guardrails, evaluation, monitoring, cost and speed. Interviewers want a clear request flow and honest trade-offs, not a list of product names.

No question matches that search. Try a shorter word.

20questions

AI system design interview questions

The full set, with labs and mock interviews, is in the AI Engineer course.

See the AI Engineer course ▸

AI System Design

1

What does a system design interview test?

Whether you ask the right questions, make sensible trade-offs, know how the parts fit together, and think about real problems like permissions, cost and failures.

2

What should you do first?

Ask questions first: who the users are, what documents, how fresh they must be, who can see what, how fast it must be, and how success is measured.

3

Why estimate scale out loud?

Numbers guide the design. For example, 200,000 documents of 5,000 tokens is about 1 billion tokens. That tells you the cost, storage and how big the system must be.

4

How do you design for latency targets, like the first word within one second?

Give each step a time budget, stream the answer, keep prompts short, cache, use smaller models for early steps, and run steps at the same time when possible.

5

Describe the high-level architecture of a RAG assistant.

Two parts. Ingestion: collect documents, read them (with OCR for scans), chunk, add details, embed, and store. Query: rewrite the question, filter by access, search, re-rank, and answer with sources.

6

How do you keep the index fresh?

Update only documents that changed, remove deleted ones, and run a full check now and then.

7

How do you control cost at scale?

Cache common answers per access group, use small models for rewriting questions, reuse shared instructions, and set cost and time limits for each step.

8

How do you handle permissions properly?

Save access groups on every chunk and filter by the user's groups, from their login, before searching. Restricted text never reaches the model.

9

What failures do you plan for?

Plan a backup for each part: if the vector database is down, use keyword search; if the model provider is down, use a backup model; treat document text as data; and date every document.

10

What trade-offs do you name at the end?

Chunk size versus precision, re-ranker quality versus speed, hosted versus self-hosted control, and how you will measure success.

11

Design a customer support agent that can issue refunds.

Classify each message, answer policy questions from documents with RAG, and give the agent a refund tool with rules enforced on the server: check the order, the amount limit and the policy. Refunds above the limit go to a human. Log everything and test on past tickets.

AI System Design: The Building Blocks

12

What does an API gateway do in front of an AI system?

It is the single front door. It checks who is calling, applies rate limits, routes the request to the right service and records it. Without it, every service would repeat the same checks.

13

When do you put a human in the loop?

When a mistake is costly or cannot be undone, such as a refund, a payment or a message to a customer. The AI prepares the action and explains why. A person approves, edits or rejects it. Low-risk actions can run on their own.

14

Where do you check identity and permissions, and why not in the prompt?

At the gateway and again in the service that owns the data. A prompt is only a request to the model, and a clever message can talk the model out of it. Code that checks the user and filters the data cannot be talked out of anything.

15

Where do guardrails sit, and what do they check?

On both sides of the model. Input guardrails mask personal data and block attacks and off-topic requests. Output guardrails check policy, check the answer against the sources and check the format. Risky cases go to a human.

16

How does a semantic cache work, and when is it unsafe?

It saves an answer with the meaning of its question. When a new question means the same thing, the saved answer is returned with no model call. It is unsafe for personal answers and for facts that change, so cache by user or tenant and set an expiry.

Solution Architecture, System Design and Technical Scoping

17

Give one non-functional requirement.

For example: answer in under 3 seconds, or 99.9% uptime.

18

Where do you check who the user is?

At the edge, before anything else runs.

19

The main model is down. What should happen?

Fall back to another model or a safe message.

20

What goes into an architecture decision record, and why write one?

Four short parts: the context, the options considered, the decision, and what you gave up. It takes ten minutes to write. Months later it tells the customer's team why the system is built this way, and stops the same debate from starting again.

More free interview questions by topic

ready to practise?

Now test yourself

Play the free quiz, or try a mock interview where you answer out loud and get a score.

Your next AI interview is closer than you think

Interview-ready. From ₹999.

Four courses: FDE, AI Agent Developer, AI Engineer and Claude Certified Architect. Narrated lessons, labs where you write the code, and every interview answer written out. Pay once and start today.

Enrol now ▸ Take the free quiz