# How Shannon's Audit Logging System Tracks Session Metrics and Per-Agent Execution Details

> Discover how Shannon's audit logging system tracks session metrics and agent execution details using a three-layer architecture for complete pentest workflow telemetry.

- Repository: [KeygraphHQ/shannon](https://github.com/keygraphhq/shannon)
- Tags: internals
- Published: 2026-02-16

---

**Shannon's audit logging system uses a three-layer architecture—`MetricsTracker` for aggregated session JSON, `AgentLogger` for append-only per-agent execution logs, and `WorkflowLogger` for human-readable real-time monitoring—to capture complete pentest workflow telemetry with crash-safe writes.**

The KeygraphHQ/shannon repository implements a comprehensive audit framework that records every step of automated security testing. This system balances machine-parseable metrics for downstream analysis with operator-friendly logs for real-time monitoring. Understanding how Shannon's audit logging system structures this data helps developers integrate custom reporting tools and debug complex multi-agent workflows.

## Architecture Overview

Shannon's audit infrastructure consists of four coordinated components:

| Component | Purpose | Output Format |
|-----------|---------|---------------|
| **`MetricsTracker`** | Aggregates timing, cost, and success data across the entire session and per-agent attempts | Structured JSON ([`session.json`](https://github.com/KeygraphHQ/shannon/blob/main/session.json)) |
| **`AgentLogger`** | Records granular execution details for individual agent runs including tool calls and LLM responses | Append-only JSON lines (`<agent>_<timestamp>_attempt_<n>.log`) |
| **`WorkflowLogger`** | Provides a consolidated human-readable timeline of phases, agents, and tools | Plain text log (`workflow.log`) |
| **`AuditSession`** | Facade that orchestrates the three loggers with mutex-protected updates to prevent race conditions | API abstraction |

## Session-Wide Metrics Tracking

The `MetricsTracker` class in [`src/audit/metrics-tracker.ts`](https://github.com/KeygraphHQ/shannon/blob/main/src/audit/metrics-tracker.ts) maintains the canonical [`session.json`](https://github.com/KeygraphHQ/shannon/blob/main/session.json) file that aggregates performance and cost metrics across all pipeline phases.

### Initialization and Agent Lifecycle

When an audit session begins, `MetricsTracker.initialize()` creates or loads the existing [`session.json`](https://github.com/KeygraphHQ/shannon/blob/main/session.json) structure. As agents execute, the tracker maintains an in-memory map of `activeTimers` to measure wall-clock duration.

To start tracking an agent, the system calls `startAgent(agentName, attemptNumber)`, which records the start timestamp in `activeTimers`. When the agent completes, `endAgent(agentName, result)` performs several critical operations:

1. Calculates duration by comparing the stored start time against the current timestamp
2. Merges attempt data including cost, success status, model name, and any errors
3. Updates per-agent aggregates and recomputes session totals
4. Writes the updated JSON atomically to prevent corruption during crashes

### Aggregation and Phase Metrics

After each agent completes, `recalculateAggregations()` walks through all successful agents to compute session-wide totals for duration and cost. The system then groups agents by pipeline phase using `AGENT_PHASE_MAP` to generate per-phase breakdowns via `calculatePhaseMetrics()`.

Each phase entry includes total duration, percentage of overall runtime, cumulative cost, and agent count. The `updateSessionStatus()` method manages high-level session state transitions between `in-progress`, `completed`, and `failed`, optionally recording the `completedAt` timestamp.

## Per-Agent Execution Logging

While `MetricsTracker` handles aggregates, the `AgentLogger` class in [`src/audit/logger.ts`](https://github.com/KeygraphHQ/shannon/blob/main/src/audit/logger.ts) captures granular execution traces for individual agent attempts.

### Append-Only JSON Structure

Each agent attempt receives a dedicated `AgentLogger` instance with a unique timestamp-based filename following the pattern `<session>/agents/<agent>_<timestamp>_attempt_<n>.log`. The constructor writes a header containing session metadata including agent name, attempt number, start time, session ID, and target URL.

During execution, `logEvent(eventType, eventData)` serializes structured data as JSON lines containing:
- **Event type** (e.g., `tool_start`, `tool_end`, `llm_response`)
- **ISO 8601 timestamp**
- **Event payload** with tool parameters, LLM content, or custom data

### Crash-Safe Write Guarantees

The `AgentLogger` implements crash-safe persistence by using Node.js write streams with explicit drain handling. Each `logEvent` call returns a Promise that resolves only when the underlying stream emits the `drain` event, ensuring data is persisted to disk before the application continues. This prevents log corruption during unexpected crashes or power failures.

For reproducibility, the first attempt of any agent also triggers `savePromptSnapshot()`, which writes the exact prompt content to a separate file for later analysis or debugging.

## Human-Readable Workflow Monitoring

The `WorkflowLogger` in [`src/audit/workflow-logger.ts`](https://github.com/KeygraphHQ/shannon/blob/main/src/audit/workflow-logger.ts) provides operators with a consolidated plain-text log suitable for real-time monitoring via `tail -f`.

The logger writes a session header containing the workflow ID, target URL, and start timestamp. As the pipeline progresses, it records:

- **Phase transitions** via `logPhase(phase, 'start'|'complete')`, producing entries like `[2026-02-16 12:00:00] [PHASE] Starting: recon`
- **Agent lifecycle** via `logAgent(agentName, 'start'|'end', details)`, printing concise status summaries including duration and cost (e.g., `sql-injection: Completed (45.2s $0.12)`)
- **Tool and LLM events** via `logToolStart` and `logLlmResponse`, formatting parameters and responses for single-line readability

When the workflow completes, `logWorkflowComplete(summary)` writes a final block containing session totals, per-agent breakdowns, and any error information.

## Coordinated Audit Facade

The `AuditSession` class in [`src/audit/audit-session.ts`](https://github.com/KeygraphHQ/shannon/blob/main/src/audit/audit-session.ts) serves as the primary interface for the rest of the codebase, coordinating the three specialized loggers while ensuring data consistency during concurrent operations.

### Mutex-Protected Updates

To prevent race conditions during parallel exploitation phases where multiple agents may complete simultaneously, `AuditSession` implements a session-scoped mutex. When `endAgent()` is called, the system acquires a lock via `sessionMutex.lock(sessionId)`, reloads the latest [`session.json`](https://github.com/KeygraphHQ/shannon/blob/main/session.json) from disk, updates metrics through `MetricsTracker`, writes the file atomically, and finally releases the lock. This guarantees that concurrent updates do not corrupt the aggregated metrics.

### Lazy Initialization

The `ensureInitialized()` method implements lazy initialization, creating the audit directory structure, initializing the `MetricsTracker`, and instantiating the `WorkflowLogger` only when first needed. This prevents unnecessary filesystem operations if the audit system is instantiated but not used.

The typical lifecycle flow involves calling `startAgent()` to begin tracking, `logEvent()` during execution for granular traces, and `endAgent()` to finalize metrics and release resources.

## Implementation Example

The following example demonstrates the complete audit lifecycle for a reconnaissance phase:

```typescript
import { AuditSession } from './src/audit/audit-session.js';
import { SessionMetadata } from './src/audit/utils.js';

async function runSecurityAudit() {
  const meta: SessionMetadata = {
    id: 'pentest-2026-02-16',
    webUrl: 'https://target.example.com',
    createdAt: new Date().toISOString(),
  };

  const audit = new AuditSession(meta);
  await audit.initialize();

  // Phase tracking
  await audit.logPhaseStart('reconnaissance');

  // Agent execution with full audit trail
  const prompt = `Enumerate subdomains for ${meta.webUrl} using passive techniques`;
  await audit.startAgent('subdomain-enum', prompt, 1);
  
  await audit.logEvent('tool_start', { 
    toolName: 'Subfinder', 
    parameters: { domain: meta.webUrl } 
  });
  
  // Simulate tool execution and LLM processing
  await audit.logEvent('llm_response', { 
    turn: 1, 
    model: 'claude-3-5-sonnet',
    content: 'Found 12 subdomains...' 
  });
  
  await audit.logEvent('tool_end', { 
    toolName: 'Subfinder', 
    findings: 12 
  });

  await audit.endAgent('subdomain-enum', {
    attemptNumber: 1,
    duration_ms: 8450,
    cost_usd: 0.08,
    success: true,
    model: 'claude-3-5-sonnet',
    isFinalAttempt: true,
  });

  await audit.logPhaseComplete('reconnaissance');

  // Retrieve aggregated metrics
  const metrics = await audit.getMetrics();
  console.log(`Total session duration: ${metrics.metrics.total_duration_ms}ms`);
  console.log(`Phase breakdown:`, metrics.metrics.phases);
}

runDemo().catch(console.error);

```

## Summary

Shannon's audit logging system provides comprehensive observability through three complementary layers:

- **`MetricsTracker`** aggregates session-wide and per-agent metrics into a structured [`session.json`](https://github.com/KeygraphHQ/shannon/blob/main/session.json), tracking duration, cost, and success rates across pipeline phases.
- **`AgentLogger`** captures granular execution traces in append-only JSON line format, ensuring crash-safe persistence of tool calls, LLM responses, and raw events.
- **`WorkflowLogger`** generates human-readable plain-text logs for real-time monitoring via standard Unix tools like `tail -f`.
- **`AuditSession`** orchestrates these components with mutex-protected updates to prevent race conditions during parallel agent execution, providing a unified API for the pentest workflow.

## Frequently Asked Questions

### How does Shannon prevent data corruption when multiple agents complete simultaneously?

Shannon implements a session-scoped mutex in `AuditSession` that locks the session ID during metric updates. When `endAgent()` is called, the system acquires the lock via `sessionMutex.lock(sessionId)`, reloads the latest [`session.json`](https://github.com/KeygraphHQ/shannon/blob/main/session.json) from disk, updates the metrics through `MetricsTracker`, writes the file atomically, and releases the lock. This guarantees that concurrent updates during parallel exploitation phases do not corrupt the aggregated session data.

### What is the difference between session.json and the per-agent log files?

[`session.json`](https://github.com/KeygraphHQ/shannon/blob/main/session.json) is a structured JSON file maintained by `MetricsTracker` that contains aggregated session-wide metrics including total duration, cumulative cost, per-phase breakdowns, and summarized per-agent statistics. In contrast, per-agent log files (created by `AgentLogger`) are append-only JSON line files that capture granular execution details for individual agent attempts, including every tool invocation, parameter set, LLM response, and raw event timestamp.

### How does the system ensure log durability during crashes?

`AgentLogger` implements crash-safe persistence by using Node.js write streams with explicit drain handling. Each `logEvent()` call returns a Promise that resolves only when the underlying stream emits the `drain` event, ensuring data is physically persisted to disk before the application continues execution. Additionally, `MetricsTracker` writes [`session.json`](https://github.com/KeygraphHQ/shannon/blob/main/session.json) atomically to prevent partial writes during system failures.

### Can operators monitor agent execution in real time?

Yes, the `WorkflowLogger` generates a human-readable `workflow.log` file specifically designed for real-time monitoring. Operators can use standard Unix tools like `tail -f` to watch phase transitions, agent start/completion events with duration and cost summaries, tool invocations, and LLM responses as they occur. The log uses single-line formatted entries with ISO timestamps for easy parsing by both humans and simple grep/sed pipelines.