WhatsApp system design: one message, from sent to read

Follow one message from DK to Rajat: WebSockets, a session cache, durable storage, and the sent, delivered and read ticks travelling back.

Both online

  1. DK sends: dinner at 8? It leaves his phone already encrypted, over a WebSocket that stays open.
  2. The load balancer keeps him on the same chat server every time, and Chat Server A gives the message an ID.
  3. Before anything else, the Message Service writes it to Cassandra, on three replicas.
  4. Only once it’s safe does DK get one tick: sent, saved on WhatsApp’s servers.
  5. Meanwhile, Chat Server A asks Presence: is Rajat online? He is.
  6. But he’s connected to a different chat server, and Redis knows which one: Chat Server B.
  7. Server A forwards it to B, which pushes it down Rajat’s open socket, and a banner appears on his phone.
  8. His phone sends back a delivery receipt, relayed the same way, and DK sees two grey ticks, before Rajat has even opened it.
  9. Rajat taps the banner and opens the chat.
  10. Opening it sends a read receipt back along the same path, and DK’s two ticks turn blue.
  11. Now Rajat replies: okay, coach.
  12. The same path in reverse: Chat Server B gives it an ID, and Cassandra stores it.
  13. Once it’s stored, Chat Server B acks, and Rajat gets his one tick: sent.
  14. Chat Server B asks Redis where DK is: Chat Server A.
  15. Server A pushes the reply down DK’s socket, through the same load balancer, the other way.
  16. DK has the chat open, so the reply is delivered and read at once, and Rajat’s tick goes straight to blue.
  17. Every tick you see is a real event, and WhatsApp’s servers never read a word.

Rajat away

  1. DK texts Rajat: dinner at 8? But Rajat’s WhatsApp is closed. Keep an eye on where the message waits.
  2. The load balancer keeps him on the same chat server every time, and Chat Server A gives the message an ID.
  3. Before anything else, the Message Service writes it to Cassandra, on three replicas.
  4. Only once it’s safe does DK get one tick: sent, saved on WhatsApp’s servers.
  5. Meanwhile, Chat Server A asks Presence: is Rajat online? No: last seen at 9:40.
  6. With no live socket to push to, the message is parked in Rajat’s inbox in Redis.
  7. The Message Service publishes an undelivered event to Kafka, for the Notification service to pick up.
  8. Notification asks Apple’s push service to wake Rajat’s phone, and a banner lights up his lock screen.
  9. The push also wakes WhatsApp in the background, just long enough to open a short socket to Chat Server B.
  10. Chat Server B drains his inbox from Redis and hands over what waited, so the message is now stored on his phone.
  11. His phone sends a delivery receipt back through the chat servers, and DK sees two grey ticks, though Rajat hasn’t looked at it.
  12. Rajat never opens it, so the ticks stay grey: delivered, not read.

Media upload

  1. DK sends Rajat a photo. Keep an eye on the upload: big files never ride the chat socket, so the photo travels on its own path first.
  2. The app asks the API gateway, and the Media Service hands back a presigned upload URL.
  3. The phone encrypts the photo and uploads it straight to blob storage: 2.4 megabytes, stored.
  4. It gets back a media ID, and that small ID is all that rides the socket.
  5. DK sends the media ID like any message, encrypted, over his open WebSocket.
  6. The load balancer keeps him on the same chat server every time, and Chat Server A gives the message an ID.
  7. Before anything else, the Message Service writes it to Cassandra, on three replicas.
  8. Only once it’s safe does DK get one tick: sent, saved on WhatsApp’s servers.
  9. Meanwhile, Chat Server A asks Presence: is Rajat online? He is.
  10. But he’s connected to a different chat server, and Redis knows which one: Chat Server B.
  11. Server A forwards it to B, which pushes it down Rajat’s open socket, and a photo banner appears on his phone.
  12. Rajat’s phone downloads the photo by itself, from the CDN.
  13. The first copy comes all the way from blob storage, and the CDN keeps it at the edge, so the next download is faster.
  14. Only now, with the photo on his phone, does it send the delivery receipt, and DK sees two grey ticks.
  15. Rajat taps the banner and opens the chat.
  16. Opening it sends a read receipt back along the same path, and DK’s two ticks turn blue.
  17. Heavy bytes through storage and the CDN, a tiny ID through the chat: that’s how a photo travels.