Design ChatGPT

You're asked to design a consumer conversational AI product. A user types a message, tokens begin appearing almost immediately, and the conversation continues over many turns with the assistant remembering what was said.

An important constraint in this system is GPU inference capacity, so we'll focus the deep dives on context management, model routing, and queueing mechanisms.

Clarify the requirements

• Is this chat, or a full agent? Confirm the scope. A chat product with occasional tool calls is different from a general agent, so we need to know which one we're designing.
• How long are conversations? Long enough to exceed the context window, which is what makes context management a required part of the design rather than an optimization.
• Do we serve the model ourselves? If inference is out of scope, treat it as a service with known latency and cost. Design an LLM Query Batching System covers its internals.
• Free and paid tiers? Yes, and that's what makes capacity allocation a design problem rather than a billing one.
• Is memory across conversations in scope? Ask explicitly because persistent memory differs from within-conversation context.

Assume: multi-turn chat with tool calls, conversations that outgrow the context window, inference as a service, tiered users, and cross-conversation memory.

Back-of-envelope numbers

• Users: $200\text{M weekly}$, $50\text{M daily}$, at $10 \text{ messages/day}$ each
• Requests: $500\text{M messages/day} \approx 6\text{k/sec}$ average, peaking near $20\text{k/sec}$
• Tokens generated: $500\text{M} \times 500 \text{ output tokens} = 250\text{B tokens/day}$
• Input tokens are larger: with history, $500\text{M} \times 3\text{k} = 1.5\text{T input tokens/day}$
• Conversation storage: $500\text{M} \times 2\text{ KB} = 1\text{ TB/day}$, which is trivial next to the compute
• Concurrent streams at $20\text{k/sec}$ arriving and $10\text{ seconds}$ each: $\approx 200\text{k open connections}$

Input volume is six times output volume, so resending conversation history dominates cost. We should treat context management as both a quality and cost decision.

High-level architecture

Components

1. API gateway. Authentication, rate limits, and tier identification.
2. Chat service. Assembles the prompt, drives tool calls, and relays the stream.
3. Conversation store. Message history, partitioned by conversation.
4. Memory store. Durable facts about the user, extracted across conversations.
5. Safety filters. Input and output classification, on the request path.
6. Inference gateway. Routes to a model tier and enforces per-user budgets.
7. Batching queue. Groups requests to keep accelerators busy.
8. GPU fleet. The scarce, expensive resource everything else exists to use well.

Deep dive 1: streaming, and why it defines the request path

The product's perceived speed comes almost entirely from streaming. A ten-second response delivered at once feels broken; the same response with tokens appearing after 500ms feels immediate. That difference is worth more than any latency optimization available elsewhere in the system.

Server-sent events are the right transport. The flow is one-directional, since the server pushes tokens and the client sends nothing mid-response, and SSE runs over plain HTTP with automatic reconnection, which avoids the operational weight of WebSockets for a channel that doesn't need bidirectionality.

Three consequences shape the rest of the design:

• Requests are long-lived, seconds to minutes. Load balancers and proxies need timeouts set accordingly, and connection count rather than request rate becomes the capacity metric for the serving tier.
• Time to first token is the latency metric that matters. Total generation time is bounded by the model's speed and is largely fixed; time to first token is what the user experiences as responsiveness, and it's what prompt assembly, queueing, and routing all affect.
• Interruption has to be handled. A user closing the tab should stop generation, or you keep paying for tokens nobody will read. Cancellation must propagate from the dropped connection all the way to the inference request. It's easy to omit, and it accounts for a meaningful fraction of wasted compute.

Partial output needs persisting. If a connection drops mid-response, the tokens already generated should be saved so reconnecting shows the partial answer rather than an empty turn.

Deep dive 2: managing context as conversations grow

Every turn resends the conversation, so a long chat means a large prompt, and prompt tokens are both the latency cost and the majority of the bill. Eventually the history exceeds the context window entirely and something must be dropped.

The approaches, in the order you'd apply them:

• Sliding window. Keep the most recent N turns verbatim. Simple, and it silently forgets the constraint the user set in their first message, which is the failure users notice.
• Summarize and carry forward. Compress older turns into a running summary, keep recent turns verbatim. Retains the thread of a long conversation at a fraction of the tokens, and costs an extra model call when the summary is refreshed. This is the workhorse.
• Retrieve from history. Embed past turns and pull back only those relevant to the current message. Effective in very long conversations where the relevant exchange was fifty turns ago and no summary would have preserved its details.
• Always pin the system prompt and the first user message, since the first message usually contains the task definition and losing it is the most damaging possible truncation.

Prompt-prefix caching is the largest cost lever available. The system prompt and the stable early history are identical across turns, and inference providers can cache the computed attention state for a shared prefix. Structuring the prompt so the stable content comes first and the variable content last, rather than interleaving them, is what makes the cache hit. Getting this ordering right can cut the per-turn cost substantially, and it's a detail that distinguishes someone who has actually built on these APIs.

Deep dive 3: allocating scarce capacity

GPU capacity is fixed on any timescale shorter than a hardware procurement cycle, so demand exceeding supply is the normal state rather than an incident. The design has to decide who gets served.

Route by difficulty, not by user. A smaller model can answer most short, easy messages at lower cost and latency. We can use a classifier to route hard messages upward, or let the small model escalate when uncertain. Since easy turns dominate traffic, this saves substantial accelerator capacity.

Tier queues, don't just tier limits. Paid users get priority admission to the inference queue rather than merely a higher message cap. Under saturation, that's the difference between paid users seeing normal latency and everyone degrading equally.

Define degradation explicitly. When capacity is short, we can show a queue, fall back to a smaller model, or reject the request. Choose one behavior for the scoped product and make any fallback visible to the user.

Cache what repeats. Exact-match caching catches surprisingly common identical prompts. Semantic caching, which matches on embedding similarity, catches near-duplicates, and needs a conservative threshold, since returning a cached answer to a subtly different question is worse than recomputing.

Budgets per user and per tier stop a single automated client from consuming a disproportionate share. These belong at the inference gateway, where a request can be rejected before it occupies an accelerator.

Deep dive 4: memory, safety, and abuse

Memory across conversations is a separate mechanism from in-conversation context, and conflating them is a common error. Persistent memory is a small set of extracted, durable facts such as preferences, ongoing projects, and stated constraints, written by a background extraction step after a conversation and retrieved into future prompts. Three properties matter: it should be small, since every fact costs tokens on every future turn; inspectable and editable, because a wrong remembered fact silently degrades every subsequent conversation; and explicitly scoped, since users are rightly sensitive about what's retained.

Safety filtering runs on both sides of the model. Input classification catches prompts that shouldn't be served, and output classification catches responses that shouldn't be shown. Output filtering interacts badly with streaming, because tokens are already on the user's screen when a violation is detected, so the practical design buffers a small window before display, or classifies incrementally and stops the stream.

Abuse is mostly automation. The economics of the product make it attractive to scrape through a consumer account rather than pay for API access, and the signals are behavioral: request rate, timing regularity, and the absence of human reading pauses between turns. Per-account rate limits combined with cost budgets are the effective control, since they cap the value of any single compromised account regardless of how it's being driven.

Common pitfalls

• Returning the complete response rather than streaming. The product feels broken even at identical total latency.
• Resending full history unmanaged. Cost and latency grow with conversation length until the window overflows.
• Interleaving stable and variable prompt content. Prefix caching can't hit, and that's the largest cost lever available.
• One model for all traffic. Most messages don't need the largest model.
• Conflating memory with context. They have different lifetimes, different sizes, and different privacy expectations.

Leveling signals

Related lessons

Design an LLM Query Batching SystemHard

Batch inference requests to maximize GPU utilization while users wait synchronously.

Design Claude CodeHard

Run a coding agent that reads and edits files and executes commands, locally and in the cloud.