Source Ledger Management in Claude-Obsidian: Architecture and Implementation

Claude-Obsidian manages source provenance through a JSON-based source ledger located at wiki/meta/ledgers/source-ledger.json that uses deterministic ID generation and strict schema validation to ensure data integrity.

Every imported source in the AgriciDaniel/claude-obsidian repository is tracked within a single source-of-truth ledger system. This component maintains comprehensive provenance records, enabling reliable cross-referencing, deduplication, and audit trails across the knowledge vault.

Source Ledger Architecture and Schema

The source ledger follows a strict contract defined by SOURCE_SCHEMA = "claude-obsidian.source-ledger.v1" implemented in claude_obsidian/ledgers.py.

Canonical File Location and Structure

The ledger persists as a JSON file at the canonical path wiki/meta/ledgers/source-ledger.json. The top-level object contains three mandatory fields:

• schema: A constant string identifying the contract version (claude-obsidian.source-ledger.v1)
• generated_at: A UTC timestamp in ISO-8601 format created by _timestamp()
• sources: A mapping object where keys represent stable source IDs and values contain source records

Core Data Structure

Each entry in the sources mapping stores provenance metadata including the origin object (with kind and locator), content_kind, authority, review_status, title, and optional content_sha256 hashes.

Stable Source ID Generation

Claude-Obsidian generates deterministic identifiers through the stable_source_id(origin_kind, locator, content_sha256) function to guarantee that identical origins always map to the same ID.

The implementation normalizes the locator, concatenates the origin kind, normalized locator, and optional content SHA-256, then hashes the result using SHA-256. The function prefixes the first 20 hexadecimal characters with src-, producing IDs matching the regex src-[A-Za-z0-9][A-Za-z0-9._-]*. This approach enables reliable deduplication and cross-referencing across the vault.

Ledger Lifecycle and Validation Pipeline

Creating a Fresh Ledger

When initializing a new vault, the system invokes empty_source_ledger(generated_at=None) to produce a compliant empty structure. This helper returns a dictionary satisfying the schema contract, ready for persistence to the canonical JSON file.

Comprehensive Validation Rules

Before any write operation, validate_source_ledger(ledger, …) enforces a rigorous invariant set defined in claude_obsidian/ledgers.py:

• Schema verification: The schema field must exactly equal SOURCE_SCHEMA
• Timestamp sanity: generated_at must be a valid UTC timestamp not exceeding the audit date
• Structural integrity: The sources field must be an object (dictionary)
• Source-ID safety: Keys must conform to the strict regex pattern
• Origin validation: The origin object must contain supported kind values (file, url, or manual) with non-empty scalar locators. URL sources require absolute https URLs without fragments and pass credential leak detection via url_credential_issue. File sources must be vault-relative and pass assert_within checks.
• Metadata constraints: Fields like content_kind, authority, review_status, and title must belong to predefined enumerations and contain only Unicode scalar values
• Content hash verification: If provided, content_sha256 must be a 64-character lowercase hexadecimal string

Error Handling with LedgerValidationError

All validation failures accumulate as structured dictionaries with { "path": "...", "message": "…" } format. The system raises these as a LedgerValidationError exception, preventing corrupted or non-compliant data from entering the canonical state.

Integration with Vault Operations

During vault adoption or upgrades, validate_existing_canonical_state(vault_root, fallback_source_ledger=None) reads the existing source-ledger.json file and applies the full validation pipeline. This function, invoked from vault_ops.py during legacy manifest migrations, either returns the validated ledger (or a fallback) or aborts the migration if integrity violations are detected.

This integration ensures that migration workflows cannot proceed with corrupted provenance data, maintaining the ledger's role as the authoritative source tracking system.

Practical Implementation Examples

The following Python code demonstrates ledger creation, source registration, and validation:

from pathlib import Path
from claude_obsidian.ledgers import (
    empty_source_ledger,
    stable_source_id,
    validate_source_ledger,
    LedgerValidationError,
)

# 1️⃣ Create a fresh ledger

ledger = empty_source_ledger()
print(ledger["schema"])                     # → claude-obsidian.source-ledger.v1

# 2️⃣ Add a new source record (normally done by the ingest skill)

origin_kind = "url"
locator = "https://example.com/article.md"
content_sha256 = "d41d8cd98f00b204e9800998ecf8427e"  # example hash

source_id = stable_source_id(origin_kind, locator, content_sha256)

ledger["sources"][source_id] = {
    "origin": {"kind": origin_kind, "locator": locator},
    "content_kind": "document",
    "authority": "official",
    "review_status": "active",
    "title": "Example Article",
    "content_sha256": content_sha256,
}

# 3️⃣ Validate before persisting

try:
    errors = validate_source_ledger(ledger, vault_root=Path("/my/vault"))
    if errors:
        print("Validation issues:", errors)
    else:
        print("Ledger is valid – ready to write")
except LedgerValidationError as exc:
    print("Fatal ledger error:", exc.errors)

Summary

• The source ledger resides at wiki/meta/ledgers/source-ledger.json and follows the claude-obsidian.source-ledger.v1 schema contract defined in claude_obsidian/ledgers.py.
• Deterministic source IDs are generated via stable_source_id() using SHA-256 hashing of normalized origin data, ensuring consistent identification across sessions.
• Strict validation through validate_source_ledger() enforces schema compliance, timestamp sanity, origin safety, and metadata integrity before any persistence operation.
• Migration safety is guaranteed by validate_existing_canonical_state() in vault_ops.py, which validates existing ledgers before adopting them into the vault workflow.
• All validation errors are captured as structured data and raised through LedgerValidationError, preventing corruption of the provenance trail.

Frequently Asked Questions

Where is the source ledger stored in a Claude-Obsidian vault?

The source ledger persists as a JSON file at the canonical relative path wiki/meta/ledgers/source-ledger.json within the vault root. This location is hardcoded as the single source of truth for all provenance metadata, as implemented in claude_obsidian/ledgers.py.

How are source IDs generated to ensure consistency?

The stable_source_id(origin_kind, locator, content_sha256) function generates deterministic identifiers by normalizing the locator, concatenating the origin kind and optional content hash, computing a SHA-256 digest, and prefixing the first 20 hexadecimal characters with src-. This guarantees that identical sources always receive identical IDs, enabling reliable deduplication.

What validation rules protect the integrity of the source ledger?

The validate_source_ledger() function enforces nine categories of invariants: schema version matching, UTC timestamp validity, object structure compliance, source-ID regex validation, origin kind enumeration checks (file/url/manual), URL safety (absolute HTTPS without fragments), file path vault-relativity, metadata enumeration constraints, and 64-character hexadecimal content hash verification.

How does the ledger integrate with vault migration workflows?

During vault adoption, validate_existing_canonical_state() reads the existing source-ledger.json and runs the full validation pipeline. Called from vault_ops.py during legacy manifest migrations, this function either returns the validated ledger, substitutes a fallback ledger if provided, or aborts the migration entirely if validation errors are present, ensuring no corrupted provenance data enters the system.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}