How Shannon's Audit Logging System Tracks Session Metrics and Per-Agent Execution Details

Shannon's audit logging system uses a three-layer architecture—MetricsTracker for aggregated session JSON, AgentLogger for append-only per-agent execution logs, and WorkflowLogger for human-readable real-time monitoring—to capture complete pentest workflow telemetry with crash-safe writes.

The KeygraphHQ/shannon repository implements a comprehensive audit framework that records every step of automated security testing. This system balances machine-parseable metrics for downstream analysis with operator-friendly logs for real-time monitoring. Understanding how Shannon's audit logging system structures this data helps developers integrate custom reporting tools and debug complex multi-agent workflows.

Architecture Overview

Shannon's audit infrastructure consists of four coordinated components:

Component	Purpose	Output Format
MetricsTracker	Aggregates timing, cost, and success data across the entire session and per-agent attempts	Structured JSON (session.json)
AgentLogger	Records granular execution details for individual agent runs including tool calls and LLM responses	Append-only JSON lines (<agent>_<timestamp>_attempt_<n>.log)
WorkflowLogger	Provides a consolidated human-readable timeline of phases, agents, and tools	Plain text log (workflow.log)
AuditSession	Facade that orchestrates the three loggers with mutex-protected updates to prevent race conditions	API abstraction

Session-Wide Metrics Tracking

The MetricsTracker class in src/audit/metrics-tracker.ts maintains the canonical session.json file that aggregates performance and cost metrics across all pipeline phases.

Initialization and Agent Lifecycle

When an audit session begins, MetricsTracker.initialize() creates or loads the existing session.json structure. As agents execute, the tracker maintains an in-memory map of activeTimers to measure wall-clock duration.

To start tracking an agent, the system calls startAgent(agentName, attemptNumber), which records the start timestamp in activeTimers. When the agent completes, endAgent(agentName, result) performs several critical operations:

1. Calculates duration by comparing the stored start time against the current timestamp
2. Merges attempt data including cost, success status, model name, and any errors
3. Updates per-agent aggregates and recomputes session totals
4. Writes the updated JSON atomically to prevent corruption during crashes

Aggregation and Phase Metrics

After each agent completes, recalculateAggregations() walks through all successful agents to compute session-wide totals for duration and cost. The system then groups agents by pipeline phase using AGENT_PHASE_MAP to generate per-phase breakdowns via calculatePhaseMetrics().

Each phase entry includes total duration, percentage of overall runtime, cumulative cost, and agent count. The updateSessionStatus() method manages high-level session state transitions between in-progress, completed, and failed, optionally recording the completedAt timestamp.

Per-Agent Execution Logging

While MetricsTracker handles aggregates, the AgentLogger class in src/audit/logger.ts captures granular execution traces for individual agent attempts.

Append-Only JSON Structure

Each agent attempt receives a dedicated AgentLogger instance with a unique timestamp-based filename following the pattern <session>/agents/<agent>_<timestamp>_attempt_<n>.log. The constructor writes a header containing session metadata including agent name, attempt number, start time, session ID, and target URL.

During execution, logEvent(eventType, eventData) serializes structured data as JSON lines containing:

• Event type (e.g., tool_start, tool_end, llm_response)
• ISO 8601 timestamp
• Event payload with tool parameters, LLM content, or custom data

Crash-Safe Write Guarantees

The AgentLogger implements crash-safe persistence by using Node.js write streams with explicit drain handling. Each logEvent call returns a Promise that resolves only when the underlying stream emits the drain event, ensuring data is persisted to disk before the application continues. This prevents log corruption during unexpected crashes or power failures.

For reproducibility, the first attempt of any agent also triggers savePromptSnapshot(), which writes the exact prompt content to a separate file for later analysis or debugging.

Human-Readable Workflow Monitoring

The WorkflowLogger in src/audit/workflow-logger.ts provides operators with a consolidated plain-text log suitable for real-time monitoring via tail -f.

The logger writes a session header containing the workflow ID, target URL, and start timestamp. As the pipeline progresses, it records:

• Phase transitions via logPhase(phase, 'start'|'complete'), producing entries like [2026-02-16 12:00:00] [PHASE] Starting: recon
• Agent lifecycle via logAgent(agentName, 'start'|'end', details), printing concise status summaries including duration and cost (e.g., sql-injection: Completed (45.2s $0.12))
• Tool and LLM events via logToolStart and logLlmResponse, formatting parameters and responses for single-line readability

When the workflow completes, logWorkflowComplete(summary) writes a final block containing session totals, per-agent breakdowns, and any error information.

Coordinated Audit Facade

The AuditSession class in src/audit/audit-session.ts serves as the primary interface for the rest of the codebase, coordinating the three specialized loggers while ensuring data consistency during concurrent operations.

Mutex-Protected Updates

To prevent race conditions during parallel exploitation phases where multiple agents may complete simultaneously, AuditSession implements a session-scoped mutex. When endAgent() is called, the system acquires a lock via sessionMutex.lock(sessionId), reloads the latest session.json from disk, updates metrics through MetricsTracker, writes the file atomically, and finally releases the lock. This guarantees that concurrent updates do not corrupt the aggregated metrics.

Lazy Initialization

The ensureInitialized() method implements lazy initialization, creating the audit directory structure, initializing the MetricsTracker, and instantiating the WorkflowLogger only when first needed. This prevents unnecessary filesystem operations if the audit system is instantiated but not used.

The typical lifecycle flow involves calling startAgent() to begin tracking, logEvent() during execution for granular traces, and endAgent() to finalize metrics and release resources.

Implementation Example

The following example demonstrates the complete audit lifecycle for a reconnaissance phase:

import { AuditSession } from './src/audit/audit-session.js';
import { SessionMetadata } from './src/audit/utils.js';

async function runSecurityAudit() {
  const meta: SessionMetadata = {
    id: 'pentest-2026-02-16',
    webUrl: 'https://target.example.com',
    createdAt: new Date().toISOString(),
  };

  const audit = new AuditSession(meta);
  await audit.initialize();

  // Phase tracking
  await audit.logPhaseStart('reconnaissance');

  // Agent execution with full audit trail
  const prompt = `Enumerate subdomains for ${meta.webUrl} using passive techniques`;
  await audit.startAgent('subdomain-enum', prompt, 1);
  
  await audit.logEvent('tool_start', { 
    toolName: 'Subfinder', 
    parameters: { domain: meta.webUrl } 
  });
  
  // Simulate tool execution and LLM processing
  await audit.logEvent('llm_response', { 
    turn: 1, 
    model: 'claude-3-5-sonnet',
    content: 'Found 12 subdomains...' 
  });
  
  await audit.logEvent('tool_end', { 
    toolName: 'Subfinder', 
    findings: 12 
  });

  await audit.endAgent('subdomain-enum', {
    attemptNumber: 1,
    duration_ms: 8450,
    cost_usd: 0.08,
    success: true,
    model: 'claude-3-5-sonnet',
    isFinalAttempt: true,
  });

  await audit.logPhaseComplete('reconnaissance');

  // Retrieve aggregated metrics
  const metrics = await audit.getMetrics();
  console.log(`Total session duration: ${metrics.metrics.total_duration_ms}ms`);
  console.log(`Phase breakdown:`, metrics.metrics.phases);
}

runDemo().catch(console.error);

Summary

Shannon's audit logging system provides comprehensive observability through three complementary layers:

• MetricsTracker aggregates session-wide and per-agent metrics into a structured session.json, tracking duration, cost, and success rates across pipeline phases.
• AgentLogger captures granular execution traces in append-only JSON line format, ensuring crash-safe persistence of tool calls, LLM responses, and raw events.
• WorkflowLogger generates human-readable plain-text logs for real-time monitoring via standard Unix tools like tail -f.
• AuditSession orchestrates these components with mutex-protected updates to prevent race conditions during parallel agent execution, providing a unified API for the pentest workflow.

Frequently Asked Questions

How does Shannon prevent data corruption when multiple agents complete simultaneously?

Shannon implements a session-scoped mutex in AuditSession that locks the session ID during metric updates. When endAgent() is called, the system acquires the lock via sessionMutex.lock(sessionId), reloads the latest session.json from disk, updates the metrics through MetricsTracker, writes the file atomically, and releases the lock. This guarantees that concurrent updates during parallel exploitation phases do not corrupt the aggregated session data.

What is the difference between session.json and the per-agent log files?

session.json is a structured JSON file maintained by MetricsTracker that contains aggregated session-wide metrics including total duration, cumulative cost, per-phase breakdowns, and summarized per-agent statistics. In contrast, per-agent log files (created by AgentLogger) are append-only JSON line files that capture granular execution details for individual agent attempts, including every tool invocation, parameter set, LLM response, and raw event timestamp.

How does the system ensure log durability during crashes?

AgentLogger implements crash-safe persistence by using Node.js write streams with explicit drain handling. Each logEvent() call returns a Promise that resolves only when the underlying stream emits the drain event, ensuring data is physically persisted to disk before the application continues execution. Additionally, MetricsTracker writes session.json atomically to prevent partial writes during system failures.

Can operators monitor agent execution in real time?

Yes, the WorkflowLogger generates a human-readable workflow.log file specifically designed for real-time monitoring. Operators can use standard Unix tools like tail -f to watch phase transitions, agent start/completion events with duration and cost summaries, tool invocations, and LLM responses as they occur. The log uses single-line formatted entries with ISO timestamps for easy parsing by both humans and simple grep/sed pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}