AI system design examples

Design a production RAG service

• Ingestion can run asynchronously through a queue. Version the index, track freshness and deletions, and test how a new version becomes visible without mixing incompatible representations.
• Query path: embed → hybrid retrieval → re-rank → prompt → STREAM the answer with citations.
• Semantic cache in front of the whole thing, scoped by tenant AND the user's document permissions: people in the same tenant may have different access. Before returning a hit, enforce current authorization and check source/index versions. Permission changes or stale data must invalidate reuse.
• Evals in CI plus reviewed online feedback. A thumbs-up is a useful signal, not a verified label. Review failures before adding them to an evaluation set.
• Multi-tenant (= one system serving many customers whose data must stay isolated): filter by tenant metadata AND applicable document ACLs IN the retrieval query. Enforce authorization in the data layer, including cache hits. NEVER rely on the prompt for data isolation.

Design an LLM serving layer

Measure input-dependent latency, output length, cost, rate limits, and provider failure behavior. There is no universal latency range or reliability ranking against other APIs.

• Latency budget per request and streaming to the client.
• Timeouts + retries with backoff and jitter (retry only transient errors).
• Define a tested fallback policy, such as another permitted model or a clearly labeled unavailable response. Cached answers need freshness and permission checks. A fallback must preserve the task's safety and quality requirements.
• Bound per-user work and queue size. A queue can smooth bursts but needs an expiry or overload response when interactive deadlines cannot be met.
• Model versioning + A/B testing at the inference layer (consider a bandit only when its assumptions fit the experiment).
• Cost observability per feature and per user.
• The SENIOR answer: describe what the user SEES in each degraded path when the provider is down or GPU capacity is thin.

Design a chatbot with memory

Sessions + message history + summarization of old turns + long-term facts store (RAG on user data) + everything from the serving layer above.