Syncing 38 million emails a month without losing a thread

Modal syncs about 38 million emails and meetings a month from Google Workspace and Microsoft 365. Each one has to land on the right people, companies and deals, exactly once, within a minute or so of being sent. When it works, nobody thinks about it. When it doesn’t, a rep opens a deal and the most important email is missing, and they stop trusting the CRM.

This is a tour of the parts that turned out to be hard.

Push first, poll as a safety net

Both providers can notify us when a mailbox changes. Gmail sends push notifications through Pub/Sub; Microsoft Graph sends change notifications to a webhook. We use both, and they are fast: the median time from an email being sent to it appearing in Modal is 14 seconds.

Notifications are not guaranteed, though. Subscriptions expire, deliveries are dropped during provider incidents, and a mailbox can quietly stop sending them. So every connected mailbox is also polled using the provider’s incremental history API, every five minutes for active mailboxes and every hour for quiet ones. Polling catches about 0.3% of messages that push missed. That sounds small until you multiply it by 38 million.

One email, many copies

The same email often reaches Modal several times. If three people on your team are on a thread with a customer, we see the message in three mailboxes, each with its own provider ID. Forwarded and BCC’d copies add more.

We deduplicate on the Message-ID header, which the sending server sets once and every copy shares. When it is missing or malformed, which happens with some older mail systems, we fall back to a hash of the sender, the sent time to the second and a normalised subject and body. Each email is stored once, with a list of the mailboxes it appeared in. That list is also what drives relationship strength.

Matching emails to records

An email address on its own tells you very little. Matching works in three steps:

1. People. Every address on the message is matched to a person record, or a new one is created if the domain belongs to a company you work with.

2. Companies. People are linked to companies by domain, with a list of about 9,000 personal and shared domains (gmail.com, outlook.com and so on) that never create a company.

3. Deals. The email is attached to open deals where those people are contacts. If there are several, it goes on all of them, and the rep can move it with one click.

Shared inboxes like sales@ or billing@ are handled differently. They are synced as their own mailbox and never count towards any single teammate’s relationships.

What the webhook looks like

Customers can subscribe to the same events we use internally. When an email is synced and matched, Modal sends a payload like this:

{
  "event": "email.synced",
  "workspace_id": "ws_kitebird",
  "email": {
    "id": "eml_7xk2p9",
    "thread_id": "thr_4mq81c",
    "direction": "inbound",
    "sent_at": "2026-09-14T08:41:12Z",
    "subject": "Re: Security review next steps"
  },
  "matched": {
    "people": ["per_8f2kq1", "per_2a9wd4"],
    "companies": ["cmp_copperline"],
    "deals": ["deal_31xv7a"]
  }
}

{
  "event": "email.synced",
  "workspace_id": "ws_kitebird",
  "email": {
    "id": "eml_7xk2p9",
    "thread_id": "thr_4mq81c",
    "direction": "inbound",
    "sent_at": "2026-09-14T08:41:12Z",
    "subject": "Re: Security review next steps"
  },
  "matched": {
    "people": ["per_8f2kq1", "per_2a9wd4"],
    "companies": ["cmp_copperline"],
    "deals": ["deal_31xv7a"]
  }
}

{
  "event": "email.synced",
  "workspace_id": "ws_kitebird",
  "email": {
    "id": "eml_7xk2p9",
    "thread_id": "thr_4mq81c",
    "direction": "inbound",
    "sent_at": "2026-09-14T08:41:12Z",
    "subject": "Re: Security review next steps"
  },
  "matched": {
    "people": ["per_8f2kq1", "per_2a9wd4"],
    "companies": ["cmp_copperline"],
    "deals": ["deal_31xv7a"]
  }
}

{
  "event": "email.synced",
  "workspace_id": "ws_kitebird",
  "email": {
    "id": "eml_7xk2p9",
    "thread_id": "thr_4mq81c",
    "direction": "inbound",
    "sent_at": "2026-09-14T08:41:12Z",
    "subject": "Re: Security review next steps"
  },
  "matched": {
    "people": ["per_8f2kq1", "per_2a9wd4"],
    "companies": ["cmp_copperline"],
    "deals": ["deal_31xv7a"]
  }
}

The body is never included. If an integration needs it, it fetches the email through the API, which checks the caller’s permissions first.

Rate limits and backfill

When a new user connects, we import up to two years of history. A heavy mailbox can hold 200,000 messages, and both providers limit how quickly we can read them. Backfill runs in a separate queue at a lower priority, newest messages first, so a new user sees last week’s emails within minutes even if their 2024 archive takes a few hours.

Live sync always wins. If a provider starts returning rate-limit errors, backfill for that workspace pauses entirely until live traffic is back to normal.

Privacy controls

Sync is opt-in per person. Each user chooses which of their mailboxes are connected and can exclude domains, labels or folders. Excluded mail is filtered before it is stored, not hidden afterwards. Admins can set workspace-wide exclusions, such as an HR domain, that individual users cannot override.

Where we are now

Over the last quarter, 99.97% of emails were attached to a record within two minutes of being sent, and duplicate emails on a timeline fell to fewer than one in a million. The remaining problems are mostly mailing lists that rewrite headers. That is next on the list.