WhatsApp API architecture: every call behind one conversation

Every call behind one chat: send and ack, session lookup, delivery and read receipts, an offline inbox with push, and encrypted media upload.

Both online

  1. Here’s one WhatsApp conversation, call by call. DK’s phone sends the message over its open WebSocket, encrypted, with a client message ID.
  2. Chat Server A gives it a time-ordered message ID, and the Message Service stores it in Cassandra, on a quorum of replicas.
  3. Only then does DK get his ack, one tick, and a retried send would be deduplicated by that client ID.
  4. Chat Server A asks Redis for Rajat’s session: he’s online, on Chat Server B.
  5. It forwards the message to Chat Server B, which pushes it down Rajat’s socket, and his phone decrypts it.
  6. Rajat’s phone sends a delivery ack. Chat Server B passes the receipt back to the origin server, Chat Server A, and DK’s ticks turn to two.
  7. When Rajat opens the chat, his phone sends a read receipt, back the same way, and DK’s ticks turn blue.
  8. Now Rajat replies, okay, coach, encrypted on his phone, up his own socket to Chat Server B.
  9. The same rule: Chat Server B stores it on a quorum first.
  10. Then Rajat gets his one tick.
  11. Chat Server B asks Redis for DK’s session: Chat Server A, online.
  12. It forwards the reply to Chat Server A, which pushes it down DK’s socket.
  13. DK’s chat is open, so one read receipt covers delivered and read, and back at the origin server, Rajat’s tick goes straight to blue.
  14. Two messages, two acks, three receipts, and not one plaintext word on any server.

Every call, in order

  1. DK → Chat Server A: send — { to: rajat, client_msg_id: c-81, ciphertext } (WebSocket)
  2. Chat Server A → Message Svc: store — { message_id: m-9001 (time-ordered, from chat-a), conv_id, ciphertext } → Cassandra · QUORUM (Internal RPC)
  3. Message Svc → Chat Server A: stored · m-9001 — durable on a quorum of replicas (reply, Internal RPC)
  4. Chat Server A → DK: ack — { c-81 → m-9001, status: sent } · a retried send dedupes on client_msg_id (reply, WebSocket)
  5. Chat Server A → Redis: GET session:rajat — is he online, and which chat server holds his socket? (Internal RPC)
  6. Redis → Chat Server A: chat-b · online (reply, Internal RPC)
  7. Chat Server A → Chat Server B: deliver — { m-9001, from: dk, ciphertext, origin: chat-a } (Internal RPC)
  8. Chat Server B → Rajat: message — { m-9001, from: dk, ciphertext, ts } · decrypted on his phone (WebSocket)
  9. Rajat → Chat Server B: delivery_ack — { m-9001 } · stored on the device (reply, WebSocket)
  10. Chat Server B → Chat Server A: receipt — { m-9001, delivered } · back to the origin server (reply, Internal RPC)
  11. Chat Server A → DK: status — { m-9001, delivered } (reply, WebSocket)
  12. Rajat → Chat Server B: read — { message_ids: [m-9001] } · he opened the chat (reply, WebSocket)
  13. Chat Server B → Chat Server A: receipt — { m-9001, read } (reply, Internal RPC)
  14. Chat Server A → DK: status — { m-9001, read } (reply, WebSocket)
  15. Rajat → Chat Server B: send — { to: dk, client_msg_id: c-17, ciphertext } · "Ok Coach", encrypted on his phone (WebSocket)
  16. Chat Server B → Message Svc: store — { message_id: m-9002 (from chat-b), conv_id, ciphertext } → Cassandra · QUORUM (Internal RPC)
  17. Message Svc → Chat Server B: stored · m-9002 (reply, Internal RPC)
  18. Chat Server B → Rajat: ack — { c-17 → m-9002, status: sent } (reply, WebSocket)
  19. Chat Server B → Redis: GET session:dk (Internal RPC)
  20. Redis → Chat Server B: chat-a · online (reply, Internal RPC)
  21. Chat Server B → Chat Server A: deliver — { m-9002, from: rajat, ciphertext, origin: chat-b } (Internal RPC)
  22. Chat Server A → DK: message — { m-9002, from: rajat, ciphertext, ts } (WebSocket)
  23. DK → Chat Server A: read — { message_ids: [m-9002] } · his chat is open: delivered and read in one receipt (reply, WebSocket)
  24. Chat Server A → Chat Server B: receipt — { m-9002, read } · back to the origin server (reply, Internal RPC)
  25. Chat Server B → Rajat: status — { m-9002, read } · straight to blue (reply, WebSocket)

Rajat away

  1. Here’s a WhatsApp message to someone who’s offline. Keep an eye on where it waits. DK’s phone sends it over its WebSocket, encrypted.
  2. Chat Server A stores it through the Message Service, in Cassandra, on a quorum of replicas.
  3. So DK gets one tick: sent.
  4. Chat Server A asks Redis for Rajat’s session. There’s none: he’s been away since 9:40.
  5. So it adds the message’s ID to Rajat’s inbox in Redis. Just an index: the message itself is already in Cassandra.
  6. It tells the Message Service the message is undelivered, and that’s published to Kafka.
  7. Notification consumes the event. It can’t read the message, so the push will carry no preview.
  8. It sends an encrypted push through Apple’s or Google’s service, and Rajat’s phone decrypts it to show the banner.
  9. The push also wakes WhatsApp in the background: it opens a socket to Chat Server B.
  10. Chat Server B reads Rajat’s inbox from Redis: one waiting message.
  11. It fetches the ciphertext from Cassandra, through the Message Service.
  12. It hands the waiting message to the phone, which decrypts it and acks delivery.
  13. The inbox entry is removed.
  14. Chat Server B looks up DK’s session: Chat Server A.
  15. The delivery receipt goes there, and DK sees two grey ticks.
  16. Rajat never opens it, so the ticks stay grey: delivered, never read.

Every call, in order

  1. DK → Chat Server A: send — { to: rajat, client_msg_id: c-81, ciphertext } (WebSocket)
  2. Chat Server A → Message Svc: store — { message_id: m-9001, conv_id, ciphertext } → Cassandra · QUORUM (Internal RPC)
  3. Message Svc → Chat Server A: stored · m-9001 (reply, Internal RPC)
  4. Chat Server A → DK: ack — { c-81 → m-9001, status: sent } (reply, WebSocket)
  5. Chat Server A → Redis: GET session:rajat (Internal RPC)
  6. Redis → Chat Server A: nil · away since 9:40 (reply, Internal RPC)
  7. Chat Server A → Redis: ZADD inbox:rajat — score: ts, member: m-9001 · an index; the message is already in Cassandra (Internal RPC)
  8. Chat Server A → Message Svc: undelivered — { m-9001, to: rajat } (Internal RPC)
  9. Message Svc → Kafka: publish message.undelivered — { to: rajat, m-9001, ciphertext } (Internal RPC)
  10. Kafka → Notification: consume — { to: rajat, m-9001, ciphertext } · servers cannot read it, so no snippet (Internal RPC)
  11. Notification → Rajat: APNs / FCM push — { encrypted payload, mutable-content: 1 } · the phone decrypts it to show the banner (REST · HTTPS)
  12. Rajat → Chat Server B: CONNECT /ws — background fetch · { user_id, device_token } (WebSocket)
  13. Chat Server B → Redis: ZRANGE inbox:rajat 0 -1 (Internal RPC)
  14. Redis → Chat Server B: [ m-9001 ] (reply, Internal RPC)
  15. Chat Server B → Message Svc: fetch — { message_ids: [m-9001] } → Cassandra (Internal RPC)
  16. Message Svc → Chat Server B: [ m-9001 · ciphertext ] (reply, Internal RPC)
  17. Chat Server B → Rajat: deliver pending — [ m-9001 ] · decrypted on the phone (WebSocket)
  18. Rajat → Chat Server B: delivery_ack — { m-9001 } (reply, WebSocket)
  19. Chat Server B → Redis: ZREM inbox:rajat — m-9001 (Internal RPC)
  20. Chat Server B → Redis: GET session:dk (Internal RPC)
  21. Redis → Chat Server B: chat-a · online (reply, Internal RPC)
  22. Chat Server B → Chat Server A: receipt — { m-9001, delivered } (Internal RPC)
  23. Chat Server A → DK: status — { m-9001, delivered } (reply, WebSocket)

Media upload

  1. Here’s a WhatsApp photo, call by call. Keep an eye on the upload: the photo never rides the chat socket. DK’s app asks the API gateway to upload 2.4 megabytes.
  2. The Media Service presigns an upload URL, good for one upload, for fifteen minutes, and returns a media ID.
  3. DK’s phone gets both: a two oh one.
  4. It encrypts the photo with a one-time media key, uploads it straight to blob storage, and gets a two hundred.
  5. Now the message goes over the WebSocket: just the media ID, the key and a thumbnail, all encrypted.
  6. It’s stored in Cassandra on a quorum of replicas, as a normal message.
  7. DK gets one tick: sent.
  8. Redis says Rajat is online, on Chat Server B.
  9. The message is forwarded there and pushed to Rajat’s phone, which decrypts the media ID, key and thumbnail.
  10. Before acking, his phone downloads the photo itself, from the CDN. It’s a cache miss, so the CDN fetches it from blob storage.
  11. The encrypted bytes come back through the CDN, which keeps a copy at the edge, and the phone decrypts them with the media key.
  12. Only now, with the photo on the phone, does it ack delivery, and the receipt brings DK two grey ticks.
  13. When Rajat opens the chat, his phone sends a read receipt, Chat Server B relays it to Chat Server A, and DK’s ticks turn blue.
  14. The heavy bytes went through storage and the CDN; only a tiny encrypted ID rode the chat.

Every call, in order

  1. DK → API Gateway: POST /api/media — { type: image/jpeg, size: 2.4MB, conv_id } (REST · HTTPS)
  2. API Gateway → Media Svc: presign — { size, type } · auth already checked at the gateway (Internal RPC)
  3. Media Svc → API Gateway: { upload_url, media_id } — PUT-only, expires in 15 min (reply, Internal RPC)
  4. API Gateway → DK: 201 Created — { upload_url, media_id } (reply, REST · HTTPS)
  5. DK → Blob Storage: PUT upload_url — 2.4MB, encrypted on the phone with a one-time media key (REST · HTTPS)
  6. Blob Storage → DK: 200 OK (reply, REST · HTTPS)
  7. DK → Chat Server A: send — { to: rajat, client_msg_id: c-82, ciphertext } · holds media_id, media key, thumbnail (WebSocket)
  8. Chat Server A → Message Svc: store — { message_id: m-9003, conv_id, ciphertext } → Cassandra · QUORUM (Internal RPC)
  9. Message Svc → Chat Server A: stored · m-9003 (reply, Internal RPC)
  10. Chat Server A → DK: ack — { c-82 → m-9003, status: sent } (reply, WebSocket)
  11. Chat Server A → Redis: GET session:rajat (Internal RPC)
  12. Redis → Chat Server A: chat-b · online (reply, Internal RPC)
  13. Chat Server A → Chat Server B: deliver — { m-9003, from: dk, ciphertext, origin: chat-a } (Internal RPC)
  14. Chat Server B → Rajat: message — { m-9003, ciphertext } · decrypts to media_id + key + thumbnail (WebSocket)
  15. Rajat → CDN: GET /media/{media_id} — auto-download, before the ack (REST · HTTPS)
  16. CDN → Blob Storage: GET object — cache miss → origin (REST · HTTPS)
  17. Blob Storage → CDN: 200 · encrypted bytes (reply, REST · HTTPS)
  18. CDN → Rajat: 200 · encrypted bytes — cached at the edge; the phone decrypts with the media key (reply, REST · HTTPS)
  19. Rajat → Chat Server B: delivery_ack — { m-9003 } · the photo is on the phone (reply, WebSocket)
  20. Chat Server B → Chat Server A: receipt — { m-9003, delivered } · back to the origin server (reply, Internal RPC)
  21. Chat Server A → DK: status — { m-9003, delivered } (reply, WebSocket)
  22. Rajat → Chat Server B: read — { message_ids: [m-9003] } (reply, WebSocket)
  23. Chat Server B → Chat Server A: receipt — { m-9003, read } (reply, Internal RPC)
  24. Chat Server A → DK: status — { m-9003, read } (reply, WebSocket)