# How to Debug Issues in Macro: A Complete Guide for Developers

> Debug Macro issues effectively. Learn to enable RUST_LOG debug, use just commands, and inspect tracing spans to resolve errors in the Macro microservices platform.

- Repository: [Macro/macro](https://github.com/macro-inc/macro)
- Tags: how-to-guide
- Published: 2026-08-14

---

**Enable `RUST_LOG=debug`, use `just` commands for testing and building, and inspect instrumented tracing spans to systematically identify and resolve errors in the Macro microservices platform.**

Macro is a large-scale, Rust-based microservices platform organized as a Cargo workspace. Built by [macro-inc/macro](https://github.com/macro-inc/macro), it handles document storage, processing, communication, and infrastructure across dozens of interconnected services. Debugging issues in Macro requires understanding its layered architecture and leveraging its built-in observability tools.

## Understanding Macro's Architecture

Macro divides functionality into four distinct layers. Knowing where a problem originates helps narrow your debugging focus.

| Layer | Primary Crates / Services | Responsibility |
|-------|---------------------------|----------------|
| **Core Storage** | `document-storage-service`, `document-cognition-service`, `search_service`, `static_file_service` | Store, process, and search documents |
| **Processing** | `convert_service`, `document-text-extractor`, `search_processing_service` | Convert formats, extract text, and index content |
| **Communication** | `email_service`, `notification_service` | Handle email ingestion, sending, and user notifications |
| **Infrastructure** | `authentication_service`, `connection_gateway`, `contacts_service` | Auth, WebSocket gateway, and contact management |

All services share supporting crates like `worker-rs-otel`, `graphql_permission`, and `graphql_notification`. External dependencies—PostgreSQL, Redis, S3, OpenSearch, and DynamoDB—are wired together exclusively through environment variables loaded via the `macro_env_var` crate, never through direct `std::env::var` calls.

## Enable Structured Tracing for Deep Visibility

Most functions in Macro are instrumented with `#[tracing::instrument]`, providing automatic span creation and field extraction. To debug issues in Macro at runtime, set the log level via environment variable:

```bash
RUST_LOG=debug just run services/search_processing_service

```

For maximum verbosity, use `trace` level:

```bash
RUST_LOG=trace,macro=trace just run services/search_processing_service

```

The instrumentation includes structured fields that appear in log output. In `services/search_processing_service/src/process/document/raw_document.rs#L195`, the `index_document` function demonstrates this pattern:

```rust
#[tracing::instrument(skip(opensearch_client, db, s3_client, document_storage_bucket, search_extractor_message),
                      fields(document_id = search_extractor_message.document_id,
                             file_type = ?search_extractor_message.file_type))]
async fn index_document(...) { ... }

```

This produces log entries with `document_id` and `file_type` automatically attached, making correlation across distributed calls straightforward.

## Run Targeted Unit and Integration Tests

Macro ships with a comprehensive test suite executed through `just` helper scripts.

Run all tests:

```bash
just test

```

Test a single crate in isolation:

```bash
just test -p email_service

```

Tests require a live PostgreSQL instance. Start dependencies with:

```bash
docker compose -f docker/docker-compose.yml up -d postgres
just setup_test_envs    # Generates .env files for test configuration

```

## Validate Database Schema After Migrations

A common source of cryptic errors stems from stale SQLx metadata. After any migration, regenerate the offline query cache:

```bash
just prepare_db

```

This updates the `.sqlx` directory used by `SQLX_OFFLINE` builds. Without this step, you may encounter "no cached data" errors during compilation even when the database is properly migrated.

## Leverage the XTASK Toolchain

The `xtask` CLI centralizes development workflows. Key commands for debugging issues in Macro include:

- **`just build`** — Build all services
- **`just build_lambdas`** — Build Lambda-specific functions
- **`just check`** — Run `cargo check` without full binary compilation (faster feedback)
- **`just run <path>`** — Execute a single service locally

Example: run the static file service in isolation:

```bash
just run services/static_file_service

```

## Inspect Runtime State via Debug Endpoints

Many services expose debug endpoints gated behind feature flags. The sync service, for instance, provides:

```bash
curl "http://localhost:8000/document/123/debug_do_kv_get/my_key"

```

This dumps internal KV entries for `document_id` 123, useful when tracing state synchronization failures.

## Follow Proper Logging Conventions

When adding instrumentation to diagnose issues, adhere to Macro's established patterns:

- Use `#[tracing::instrument(err)]` for functions returning `Result` to capture error branches automatically
- Log errors with `error=?e` pattern for structured extraction
- Prefer `debug!` for verbose output toggled via `RUST_LOG`

Example from `services/email_service/src/util/sync_contacts.rs#L115`:

```rust
tracing::debug!(error=?e, link_id=%link.id, "Failed to get primary contacts");

```

The `%` sigil indicates `Display` formatting, while `?` uses `Debug` formatting—choose based on the type's implementation.

## Common Pitfalls and Solutions

### Missing Environment Variables

All secrets must flow through `macro_env_var`. Absent variables cause immediate panics at startup. Verify your `.env` files are generated correctly:

```bash
just setup_test_envs    # For test configuration

just setup_macrodb      # For local development database

```

### Incorrect SQLx Offline Usage

Do **not** run `cargo test` with `SQLX_OFFLINE=true`. Offline mode benefits only `cargo check` and `cargo build`. Tests require live database connectivity to verify query correctness.

### Stale Migration State

After editing migrations, refresh the entire schema:

```bash
just setup_macrodb

```

In rare cases requiring complete reset:

```bash
just force_drop_db

```

## Manual Tracing for Custom Investigation

When automatic instrumentation is insufficient, add spans manually:

```rust
use tracing::{debug_span, Instrument};

let span = debug_span!("process_document", doc_id = %doc.id);
async {
    // ... async work ...
}.instrument(span).await;

```

This pattern integrates seamlessly with Macro's existing `tracing` infrastructure.

## Summary

- **Enable visibility** with `RUST_LOG=debug` or `trace` and structured `tracing` spans
- **Isolate problems** using `just test -p <crate>` for targeted testing
- **Keep schema current** by running `just prepare_db` after any migration
- **Use xtask workflows** for consistent build, check, and run operations
- **Query debug endpoints** for runtime state inspection when available
- **Avoid `SQLX_OFFLINE=true`** during test execution; reserve it for compilation only

## Frequently Asked Questions

### Why does my service panic immediately on startup?

Macro services validate all required environment variables through `macro_env_var` at initialization. A missing variable triggers a panic with a clear message indicating which key is absent. Run `just setup_test_envs` or `just setup_macrodb` to ensure `.env` files are properly generated, then verify the specific variables listed in your service's configuration.

### How do I add custom debug logging to track a specific issue?

Use `tracing::debug!` with structured fields for searchable output. For function-level instrumentation, add `#[tracing::instrument(skip(...), fields(...))]` above async functions, matching the pattern in [`services/search_processing_service/src/process/document/raw_document.rs`](https://github.com/macro-inc/macro/blob/main/services/search_processing_service/src/process/document/raw_document.rs). Run with `RUST_LOG=debug` to see your output without recompiling.

### What causes "no cached data" errors during builds?

This occurs when `SQLX_OFFLINE` is enabled but the `.sqlx` query cache is stale or missing. Run `just prepare_db` after any database migration to regenerate cached query data. The cache allows compilation without database connectivity, but only if it reflects the current schema state.

### Can I debug production issues with the same tools?

Production deployments typically run at `RUST_LOG=info` or `warn` for performance. Macro's `worker-rs-otel` crate exports distributed traces to OpenTelemetry collectors. For urgent issues, temporarily elevate log levels via environment configuration, or query debug endpoints if they remain exposed behind authentication.