How to Debug Issues in Macro: A Complete Guide for Developers

Enable RUST_LOG=debug, use just commands for testing and building, and inspect instrumented tracing spans to systematically identify and resolve errors in the Macro microservices platform.

Macro is a large-scale, Rust-based microservices platform organized as a Cargo workspace. Built by macro-inc/macro, it handles document storage, processing, communication, and infrastructure across dozens of interconnected services. Debugging issues in Macro requires understanding its layered architecture and leveraging its built-in observability tools.

Understanding Macro's Architecture

Macro divides functionality into four distinct layers. Knowing where a problem originates helps narrow your debugging focus.

Layer	Primary Crates / Services	Responsibility
Core Storage	document-storage-service, document-cognition-service, search_service, static_file_service	Store, process, and search documents
Processing	convert_service, document-text-extractor, search_processing_service	Convert formats, extract text, and index content
Communication	email_service, notification_service	Handle email ingestion, sending, and user notifications
Infrastructure	authentication_service, connection_gateway, contacts_service	Auth, WebSocket gateway, and contact management

All services share supporting crates like worker-rs-otel, graphql_permission, and graphql_notification. External dependencies—PostgreSQL, Redis, S3, OpenSearch, and DynamoDB—are wired together exclusively through environment variables loaded via the macro_env_var crate, never through direct std::env::var calls.

Enable Structured Tracing for Deep Visibility

Most functions in Macro are instrumented with #[tracing::instrument], providing automatic span creation and field extraction. To debug issues in Macro at runtime, set the log level via environment variable:

RUST_LOG=debug just run services/search_processing_service

For maximum verbosity, use trace level:

RUST_LOG=trace,macro=trace just run services/search_processing_service

The instrumentation includes structured fields that appear in log output. In services/search_processing_service/src/process/document/raw_document.rs#L195, the index_document function demonstrates this pattern:

#[tracing::instrument(skip(opensearch_client, db, s3_client, document_storage_bucket, search_extractor_message),
                      fields(document_id = search_extractor_message.document_id,
                             file_type = ?search_extractor_message.file_type))]
async fn index_document(...) { ... }

This produces log entries with document_id and file_type automatically attached, making correlation across distributed calls straightforward.

Run Targeted Unit and Integration Tests

Macro ships with a comprehensive test suite executed through just helper scripts.

Run all tests:

just test

Test a single crate in isolation:

just test -p email_service

Tests require a live PostgreSQL instance. Start dependencies with:

docker compose -f docker/docker-compose.yml up -d postgres
just setup_test_envs    # Generates .env files for test configuration

Validate Database Schema After Migrations

A common source of cryptic errors stems from stale SQLx metadata. After any migration, regenerate the offline query cache:

just prepare_db

This updates the .sqlx directory used by SQLX_OFFLINE builds. Without this step, you may encounter "no cached data" errors during compilation even when the database is properly migrated.

Leverage the XTASK Toolchain

The xtask CLI centralizes development workflows. Key commands for debugging issues in Macro include:

• just build — Build all services
• just build_lambdas — Build Lambda-specific functions
• just check — Run cargo check without full binary compilation (faster feedback)
• just run <path> — Execute a single service locally

Example: run the static file service in isolation:

just run services/static_file_service

Inspect Runtime State via Debug Endpoints

Many services expose debug endpoints gated behind feature flags. The sync service, for instance, provides:

curl "http://localhost:8000/document/123/debug_do_kv_get/my_key"

This dumps internal KV entries for document_id 123, useful when tracing state synchronization failures.

Follow Proper Logging Conventions

When adding instrumentation to diagnose issues, adhere to Macro's established patterns:

• Use #[tracing::instrument(err)] for functions returning Result to capture error branches automatically
• Log errors with error=?e pattern for structured extraction
• Prefer debug! for verbose output toggled via RUST_LOG

Example from services/email_service/src/util/sync_contacts.rs#L115:

tracing::debug!(error=?e, link_id=%link.id, "Failed to get primary contacts");

The % sigil indicates Display formatting, while ? uses Debug formatting—choose based on the type's implementation.

Common Pitfalls and Solutions

Missing Environment Variables

All secrets must flow through macro_env_var. Absent variables cause immediate panics at startup. Verify your .env files are generated correctly:

just setup_test_envs    # For test configuration

just setup_macrodb      # For local development database

Incorrect SQLx Offline Usage

Do not run cargo test with SQLX_OFFLINE=true. Offline mode benefits only cargo check and cargo build. Tests require live database connectivity to verify query correctness.

Stale Migration State

After editing migrations, refresh the entire schema:

just setup_macrodb

In rare cases requiring complete reset:

just force_drop_db

Manual Tracing for Custom Investigation

When automatic instrumentation is insufficient, add spans manually:

use tracing::{debug_span, Instrument};

let span = debug_span!("process_document", doc_id = %doc.id);
async {
    // ... async work ...
}.instrument(span).await;

This pattern integrates seamlessly with Macro's existing tracing infrastructure.

Summary

• Enable visibility with RUST_LOG=debug or trace and structured tracing spans
• Isolate problems using just test -p <crate> for targeted testing
• Keep schema current by running just prepare_db after any migration
• Use xtask workflows for consistent build, check, and run operations
• Query debug endpoints for runtime state inspection when available
• Avoid SQLX_OFFLINE=true during test execution; reserve it for compilation only

Frequently Asked Questions

Why does my service panic immediately on startup?

Macro services validate all required environment variables through macro_env_var at initialization. A missing variable triggers a panic with a clear message indicating which key is absent. Run just setup_test_envs or just setup_macrodb to ensure .env files are properly generated, then verify the specific variables listed in your service's configuration.

How do I add custom debug logging to track a specific issue?

Use tracing::debug! with structured fields for searchable output. For function-level instrumentation, add #[tracing::instrument(skip(...), fields(...))] above async functions, matching the pattern in services/search_processing_service/src/process/document/raw_document.rs. Run with RUST_LOG=debug to see your output without recompiling.

What causes "no cached data" errors during builds?

This occurs when SQLX_OFFLINE is enabled but the .sqlx query cache is stale or missing. Run just prepare_db after any database migration to regenerate cached query data. The cache allows compilation without database connectivity, but only if it reflects the current schema state.

Can I debug production issues with the same tools?

Production deployments typically run at RUST_LOG=info or warn for performance. Macro's worker-rs-otel crate exports distributed traces to OpenTelemetry collectors. For urgent issues, temporarily elevate log levels via environment configuration, or query debug endpoints if they remain exposed behind authentication.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}