How the Macro Email Service Handles Multi-Account Gmail Synchronization

The Macro email service synchronizes multiple Gmail accounts per user through OAuth-linked accounts with unique link_id identifiers, Google Watch subscriptions for real-time notifications, and parallel Pub/Sub workers that fetch and store messages, contacts, and attachments.

The Macro email service is designed from the ground up to support multi-account Gmail synchronization, allowing a single user to connect and sync many Gmail accounts simultaneously without performance degradation or interference between accounts. This article examines the architecture and implementation details directly from the macro-inc/macro source code.

Account Linking and OAuth Token Management

Every synchronized Gmail account begins with a link—a persistent association between a Macro user and a specific Gmail account. The enable_gmail_sync_for helper in services/email_service/src/api/email/init.rs creates this link and securely stores the OAuth refresh token:

// Conceptual flow from init.rs lines 389+
// enable_gmail_sync_for creates link record with:
// - user_id (Macro user)
// - gmail_address 
// - refresh_token (OAuth)
// - link_id (unique identifier for this account link)

The link_id becomes the central key that isolates all per-account operations. When a worker needs to interact with Gmail, it provides this identifier to retrieve the correct credentials.

Centralized Token State

At service startup in services/email_service/src/main.rs (around line 200), the system constructs GmailTokenState—a shared cache of GmailTokenProviderImpl instances keyed by link_id:

// From main.rs – GmailTokenState construction
let gmail_token_state = Arc::new(GmailTokenState::new(
    token_providers, // HashMap<link_id, GmailTokenProviderImpl>
));

This design avoids redundant token refresh operations. Multiple workers can reference the same underlying token provider for an account, and the provider handles OAuth refresh transparently when tokens expire.

Google Watch Subscriptions for Real-Time Sync

The service does not poll Gmail. Instead, it leverages Google's push notification API through the events.watch endpoint. The watch subscription lifecycle is managed in services/email_service/src/pubsub/link_manager/process.rs (around line 160 in the "Refresh operation"):

• Creates or renews a watch for each linked account
• Stores subscription details (expiration, resource ID) with the link record
• Automatically refreshes watches before they expire

When new mail arrives in any linked Gmail account, Google publishes a message to Macro's configured Pub/Sub topic. This fan-out mechanism enables near-real-time synchronization across hundreds or thousands of accounts without excessive API quota consumption.

Parallel Inbox Synchronization Workers

The inbox_sync::worker in services/email_service/src/pubsub/inbox_sync/worker.rs forms the core synchronization engine. This worker:

1. Consumes watch notification messages from Pub/Sub
2. Extracts the link_id to identify which account changed
3. Retrieves a fresh token from GmailTokenState
4. Calls Gmail's listMessages or listThreads with appropriate filters
5. Upserts the fetched data into Macro's internal email_db

Critically, workers run in parallel for every active link. The architecture scales horizontally—additional worker instances can process the same topic, and each message carries the link_id ensuring correct account isolation.

Contact Synchronization Pipeline

Contacts are handled separately from messages via sync_contacts in services/email_service/src/util/sync_contacts.rs. This function implements incremental synchronization using per-account sync tokens:

// Conceptual flow from sync_contacts.rs
pub async fn sync_contacts(db: &Db, link_id: &str, token_state: &GmailTokenState) {
    // 1. Retrieve stored sync_token for this link_id from database
    let sync_token = db.get_contact_sync_token(link_id).await;
    
    // 2. Call Gmail People API with token (returns only changes since last sync)
    let response = people_api.list_connections()
        .sync_token(sync_token)
        .page_token(page_token)
        .await;
    
    // 3. Upsert contacts and store new sync_token for next iteration
    db.upsert_contacts(link_id, response.connections).await;
    db.update_contact_sync_token(link_id, response.next_sync_token).await;
}

If a sync token expires, the function implements fallback logic to perform a full resynchronization, then establishes a fresh incremental token.

Attachment Streaming and Storage

When messages contain attachments, the upload_attachment helper in services/email_service/src/util/upload_attachment.rs handles retrieval and storage:

1. Fetches raw attachment data from Gmail using the per-account token
2. Streams bytes directly to Macro's S3-backed storage
3. Generates a presigned URL
4. Attaches the URL reference to the message record

This approach avoids storing binary data in the primary database while maintaining security through time-limited, revocable access URLs.

Data Consistency Through Backfill and Cleanup

Two additional worker types ensure eventual consistency across all linked accounts:

Worker	Location	Purpose
Backfill worker	src/pubsub/backfill/worker.rs	Periodically scans for missed threads or contacts; imports any gaps detected
CRM cleanup worker	src/pubsub/crm_cleanup/worker.rs	Removes stale data when a Gmail account is disconnected

Both operate per-link_id, maintaining isolation guarantees. The backfill worker is particularly important for recovering from missed Pub/Sub messages or temporary Gmail API outages.

Summary

• Account isolation via unique link_id identifiers enables safe multi-account operation
• GmailTokenState centralizes OAuth management while keeping credentials scoped per account
• Google Watch subscriptions push notifications in real-time without polling overhead
• Parallel workers process inbox changes, contacts, and attachments simultaneously across all linked accounts
• Periodic backfill guarantees data completeness even when notifications fail

The Macro email service achieves scalable multi-account Gmail synchronization through careful separation of per-account state and efficient shared infrastructure for token management and event distribution.

Frequently Asked Questions

How does the service handle OAuth token expiration?

The GmailTokenProviderImpl caches tokens and automatically refreshes them using the stored refresh token when API calls return 401 errors. The GmailTokenState ensures all workers for a given link_id share the same provider instance, preventing race conditions during refresh.

What happens if a Google Watch subscription expires?

The link manager process in src/pubsub/link_manager/process.rs monitors subscription expiration times and proactively refreshes watches before they lapse. If a watch does expire, the backfill worker eventually catches any missed messages once the subscription is re-established.

Can users sync more than one Gmail account simultaneously?

Yes. Every linked account receives its own link_id, watch subscription, and parallel worker allocation. The architecture imposes no hard limit on accounts per user; practical limits depend on Gmail API quota and infrastructure scaling.

How are attachments secured after upload?

Attachments stream directly from Gmail to S3 without persisting on local filesystems. The generated presigned URLs include time-limited access tokens, and the Macro application validates user permissions before serving attachment content.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}