ChatGPT API architecture: every call behind one prompt

Every call behind one ChatGPT prompt: the JWT check, history, generation on GPUs, the SSE token stream and the save, as a sequence diagram.

Send a message

  1. Here’s your message, call by call. Your app sends one request: POST /messages, with the conversation ID and your text.
  2. The API gateway doesn’t trust it yet. It asks Auth to validate your token, and Auth answers: user 42.
  3. Then the gateway hands the message to the Message Service, with your user ID attached.
  4. The Message Service runs its own checks: 38 messages left this hour, and the prompt is safe.
  5. It loads the conversation from the database: twelve earlier messages, so the model sees the whole thread.
  6. Now it calls the LLM gateway: generate, with the model, every message, and stream set to true.
  7. The GPU workers run prefill: they read the whole prompt once, and keep its attention in the KV cache.
  8. Decode begins. The first token, “Load”, goes from the GPUs to the LLM gateway.
  9. The gateway passes it on to the Message Service, which sends it down to your phone as a server-sent event.
  10. That loop runs once for every token, so the answer appears word by word while it’s still being written.
  11. After 96 tokens, the GPUs report done, and the gateway tells the Message Service the stream is complete.
  12. Only now does the Message Service save the whole answer to the database, as message 789.
  13. The last event, done, carries that ID to your phone: the answer is saved.
  14. Seventeen calls, and your phone made only one of them: everything else came back on one open stream.

Every call, in order

  1. You → API Gateway: POST /messages — { conversation_id, message: "Explain…" } (REST · HTTPS)
  2. API Gateway → Auth: validate JWT (Internal RPC)
  3. Auth → API Gateway: ok · user_42 (reply, Internal RPC)
  4. API Gateway → Message Service: handle message — { user_42, conversation_id, text } (Internal RPC)
  5. Message Service → Message Service: rate limit · moderate — 38 left this hour · prompt is safe (Internal RPC)
  6. Message Service → Database: load history (Internal RPC)
  7. Database → Message Service: 12 messages (reply, Internal RPC)
  8. Message Service → LLM Gateway: generate — { model, messages, stream: true } (Internal RPC)
  9. LLM Gateway → GPU Workers: prefill — read the whole prompt once · keep the KV cache (Internal RPC)
  10. GPU Workers → LLM Gateway: token — { "Load" } (reply, Internal RPC)
  11. LLM Gateway → Message Service: token — { "Load" } (reply, Internal RPC)
  12. Message Service → You: event: token — data: {"content":"Load"} (reply, SSE stream)
  13. GPU Workers → LLM Gateway: done — 96 tokens (reply, Internal RPC)
  14. LLM Gateway → Message Service: stream complete (reply, Internal RPC)
  15. Message Service → Database: save message (Internal RPC)
  16. Database → Message Service: saved · msg_789 (reply, Internal RPC)
  17. Message Service → You: event: done — data: {"message_id":"msg_789"} (reply, SSE stream)