AI SYSTEM DESIGN (expect at least one)

Design a production RAG service

• Ingestion is ASYNC (queue) — docs change, the index updates without downtime. Version the index.
• Query path: embed → hybrid retrieval → re-rank → prompt → STREAM the answer with citations.
• Semantic cache in front of the whole thing.
• Evals in CI + online feedback loop (thumbs up/down feeds the golden dataset).
• Multi-tenant (= one system serving many customers whose data must stay isolated): filter by tenant metadata IN the vector query. NEVER rely on the prompt for data isolation.

Design an LLM serving layer

The special constraints: calls are SLOW (500ms-10s), EXPENSIVE, and fail MORE than normal APIs.

• Latency budget per request; streaming to the client.
• Timeouts + retries with backoff and jitter (retry only transient errors).
• FALLBACK CHAIN: primary model → smaller model → cached/static answer. Circuit breaker per provider.
• Rate limit per user + a queue to absorb bursts.
• Model versioning + A/B testing at the inference layer (bandit algorithm — you have literally done this).
• Cost observability per feature and per user.
• The SENIOR answer: describe what the user SEES in each degraded path when the provider is down or GPU capacity is thin.

Design a chatbot with memory

Sessions + message history + summarization of old turns + long-term facts store (RAG on user data) + everything from the serving layer above.