ChatGPT system design: one prompt, from your phone to the GPUs and back

Send a message

1. You press send, and your message joins thousands of others arriving in the same second. A load balancer spreads that traffic across servers.
2. A firewall drops anything that looks like an attack, and the API gateway finds the message service.
3. This orchestrator runs four checks before any AI is involved.
4. First: are you signed in? Auth validates your token.
5. Second: have you sent too many?
6. A counter in Redis keeps score, so no one user can flood the GPUs.
7. Third: is the prompt safe to answer? Moderation screens it before it ever reaches the model.
8. Fourth: your chat history, read from the database, so the model remembers the thread.
9. Notice: not one GPU has touched your message yet.
10. Only now does the AI start. A gateway picks the model, builds the input, and a scheduler batches it.
11. The GPUs read your whole prompt in one pass, the prefill, and keep it in a KV cache.
12. Then decode: they write the answer one token at a time, each step reading that cache instead of starting over.
13. It doesn’t wait for the end. Each token comes back over one open connection, server-sent events, so you watch the answer appear while it’s still being written.
14. When it’s finished, the exchange is saved to the database, and copied to read replicas for the next time you open the app.
15. Last, a done event tells your phone the conversation is safe.
16. Seventeen systems, and in real life, a few seconds.