How Gmail Watch Pub/Sub Push Works in gogcli: A Deep Dive into the `watch serve` Command

The gog gmail watch serve command starts an HTTP server that receives Gmail Pub/Sub push notifications, validates them, fetches new messages via the Gmail API, and forwards structured JSON payloads to a downstream webhook.

The steipete/gogcli tool implements a complete Gmail watch workflow using Google Cloud Pub/Sub. While gog gmail watch start registers the watch with Google, the watch serve command handles the critical push delivery mechanism. This article examines the internal architecture of the Pub/Sub push handler, from request validation to message forwarding.

Prerequisites: Registering a Gmail Watch with watch start

Before running watch serve, you must establish a watch subscription. The gog gmail watch start command registers a push notification subscription with the Gmail API and persists the configuration to disk.

In internal/cmd/gmail_watch_cmds.go, the requestGmailWatch function constructs a gmail.WatchRequest containing the Pub/Sub topic and optional label filters. The buildWatchState function then serializes this configuration—along with the returned historyId, expiration timestamp, and optional webhook URL—to a JSON state file stored at ~/.config/gogcli/state/gmail-watch/<account>.json.

This state file serves as the source of truth for the watch serve command, enabling it to resume operations across restarts without re-registering the watch.

The watch serve Architecture

When you invoke gog gmail watch serve, the tool loads the stored watch state and starts a minimal HTTP server designed specifically to handle Pub/Sub push deliveries.

Loading Watch State and Configuration

The Run function in internal/cmd/gmail_watch_cmds.go first invokes loadGmailWatchStore(account) to deserialize the persisted state. Command-line flags such as --hook-url override values stored in the state file, allowing you to update the webhook destination without restarting the watch registration.

Starting the HTTP Server

The server initialization creates a gmailWatchServer struct configured with the watch store, an optional OIDC validator, and an http.Client for outbound webhook calls. The server binds to the address specified by --bind and --port (defaulting to 127.0.0.1:8080) and begins listening for POST requests at the configured path (default /).

Handling Pub/Sub Push Messages

Each incoming push request triggers a multi-stage pipeline within the ServeHTTP method of internal/cmd/gmail_watch_server.go.

Request Authorization (OIDC vs. Shared Token)

The authorize function implements two authentication strategies. If --verify-oidc is enabled, the server extracts the JWT from the Authorization: Bearer header and validates it via verifyOIDCToken. For development or private deployments, the server accepts a shared secret via the x-gog-token header or ?token= query parameter. If neither verification method is configured, the server accepts all requests.

Decoding the Pub/Sub Envelope

Once authorized, parsePubSubPush reads the request body (capped at defaultPushBodyLimitBytes) and unmarshals the Pub/Sub JSON envelope. The message.data field contains a base64-encoded Gmail push payload, which decodeGmailPushPayload decodes and unmarshals into a gmailPushPayload struct containing the historyId, emailAddress, and messageId.

Deduplication and Validation

The server performs critical sanity checks before processing. If the incoming messageId matches LastPushMessageID stored in the state file, the push is ignored as a duplicate. Additionally, if the payload contains an emailAddress, it must match the configured account to prevent cross-account confusion.

Fetching and Processing Gmail Messages

After validation, the server uses the Gmail API to retrieve the actual message content that triggered the push notification.

Collecting Message IDs via History API

The collectHistoryMessageIDs function calls the Gmail History API:

svc.Users.History.List("me").StartHistoryId(startID).MaxResults(cfg.HistoryMax)

This retrieves a list of message IDs that changed since the last known historyId. The server determines the starting history ID via store.StartHistoryID(payload.HistoryID), which handles cases where the stored ID is missing or stale.

Resyncing on Stale History

If the History API returns a "history not found" or invalid history ID error, isStaleHistoryError detects the condition and triggers resyncHistory. This fallback performs a direct Messages.List call (limited by cfg.ResyncMax) to retrieve recent messages and resets the stored history ID to the current push's HistoryID, ensuring the watch can recover from Gmail's 7-day history expiration window.

Filtering Labels and Fetching Content

The fetchMessages function retrieves full message details for each collected ID, applying label exclusion filters (defaulting to SPAM,TRASH). If --include-body is enabled, the server extracts the text/plain content up to --max-bytes, ensuring the downstream webhook receives manageable payloads.

Webhook Delivery and State Management

Once messages are fetched and filtered, the server prepares and forwards the data.

Building the Hook Payload

The server constructs a gmailHookPayload containing the source identifier, account email, historyId, and an array of gmailHookMessage structs. Each message includes id, threadId, key headers (From, To, Subject), the snippet, and optionally the full body text.

Forwarding to Downstream Webhooks

If cfg.HookURL is configured, sendHook POSTs the JSON payload to the specified endpoint. The function records the HTTP response status and timestamp in the watch state (LastDeliveryStatus, LastDeliveryAtMs). Importantly, webhook failures do not block the watch processing; the server returns success to Pub/Sub even if the downstream hook is temporarily unavailable, relying on Gmail's eventual consistency for message recovery.

Persisting Watch State

After successful processing, the server updates the state file with the new HistoryID, LastPushMessageID, and processing timestamps. This idempotent handling ensures that duplicate Pub/Sub deliveries (common in distributed systems) do not result in duplicate webhook calls or message processing.

Summary

• gog gmail watch start registers a Gmail push notification subscription and persists watch state (topic, labels, history ID) to ~/.config/gogcli/state/gmail-watch/<account>.json.
• gog gmail watch serve launches an HTTP server that receives Pub/Sub push messages at a configurable endpoint, supporting both OIDC JWT and shared-token authentication.
• The server decodes Pub/Sub envelopes via parsePubSubPush, validates requests against duplicates and account mismatches, and uses the Gmail History API (collectHistoryMessageIDs) to fetch new messages.
• Stale history IDs trigger automatic resync via resyncHistory, which falls back to Messages.List to recover from Gmail's 7-day history expiration.
• Fetched messages are filtered by labels (excluding SPAM/TRASH by default), optionally include body content, and are forwarded as JSON to a downstream webhook via sendHook.
• The server maintains idempotent processing by tracking LastPushMessageID and HistoryID in the state file, ensuring duplicate Pub/Sub deliveries do not trigger duplicate webhooks.

Frequently Asked Questions

How do I authenticate Pub/Sub push requests without configuring OIDC?

You can use a shared secret token by passing --token <secret> when running gog gmail watch serve. The server validates this against the x-gog-token header or ?token= query parameter in incoming requests. This method is suitable for development or when running behind a reverse proxy that handles TLS termination, though OIDC JWT verification is recommended for production deployments.

What happens when the Gmail history ID becomes invalid or stale?

If the Gmail API returns a "history not found" error (typically after 7 days of inactivity), the server detects this via isStaleHistoryError and triggers resyncHistory. This fallback mechanism performs a direct Messages.List API call limited by --resync-max (defaulting to recent messages) and resets the stored history ID to the current push's history ID. This ensures the watch can recover automatically without manual intervention.

Can I filter specific labels from triggering webhook notifications?

Yes, the fetchMessages function applies label exclusion filters during message processing. By default, messages labeled SPAM or TRASH are excluded. You can customize this behavior using the --exclude-labels flag when running watch serve, passing a comma-separated list of label names to ignore. Additionally, the watch start command accepts --label filters to limit which Gmail labels trigger push notifications at the API level.

Where does gogcli store the watch state and configuration?

The watch state is persisted as JSON files in ~/.config/gogcli/state/gmail-watch/<account>.json (or the platform-specific equivalent). This file contains the Pub/Sub topic, label filters, current history ID, expiration timestamp, last push message ID, and optional webhook configuration. The watch serve command reads this state on startup and updates it after processing each push notification to maintain idempotency and track delivery status.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}