ChatDev Workspace Artifact Management and File Storage: Hook and Store Architecture

ChatDev’s artifact system combines an incremental snapshot-diff hook with a deduplicating attachment store to automatically track, persist, and stream workspace files as multimodal message blocks across workflow nodes.

OpenBMB/ChatDev isolates every node execution in a temporary workspace, requiring a robust bridge between filesystem changes and LLM consumption. The repository implements a two-layer workspace artifact management system that detects modifications via the WorkspaceArtifactHook and persists them through the AttachmentStore, enabling seamless file storage and retrieval across complex AI-driven development workflows.

How the Workspace Artifact Hook Detects File Changes

The WorkspaceArtifactHook in workflow/hooks/workspace_artifact.py acts as a filesystem watcher that snapshots a node’s workspace before and after execution. It compares these snapshots to generate WorkspaceArtifact objects representing every created, updated, or deleted file.

Snapshotting Before and After Node Execution

When the workflow executor prepares to run a node, the hook’s before_node() method captures the current workspace state in self._snapshots. This dictionary maps file paths to _FileSignature objects containing metadata and SHA-256 hashes.

# workflow/hooks/workspace_artifact.py

class WorkspaceArtifactHook:
    def __init__(..., node_types=("python","agent"), exclude_dirs=("attachments","__pycache__")):
        self._snapshots: Dict[str, Dict[str, _FileSignature]] = {}
        self._last_emitted: Dict[str, _TrackedEntry] = {}

The _snapshot() method walks the directory using os.walk, respecting configurable limits (max_files_scanned and max_bytes_scanned) to prevent performance degradation on large workspaces. It automatically skips directories listed in exclude_dirs such as __pycache__ or temp.

After the node finishes executing, after_node() generates a second snapshot and performs a diff against the first. Any path whose SHA-256 hash changed—or exists in the after snapshot but not the before—is flagged as created or updated. Conversely, paths present in _last_emitted but missing from the current snapshot are marked as deleted.

Registration and Emission

For each detected change, the hook calls _register_artifact(), which forwards the file to the AttachmentStore and constructs a WorkspaceArtifact data class containing:

• node_id and attachment_id
• Relative path, MIME type, file size, and SHA-256 hash
• change_type enum (created, updated, deleted)
• Optional extra metadata payload

All artifacts are batched and emitted via the emit_callback supplied during hook initialization, allowing the workflow runtime to stream changes to the UI or LLM immediately.

The Attachment Store – Persistent File Management

While the hook detects changes, the AttachmentStore in utils/attachments.py handles durable persistence, deduplication, and retrieval. It maintains a filesystem-backed repository where files are stored by content hash and referenced via immutable UUIDs.

# utils/attachments.py

class AttachmentStore:
    def __init__(self, root_dir: Path, inline_size_limit=DEFAULT_INLINE_LIMIT):
        self.root = Path(root_dir)
        self.manifest_path = self.root / "attachments_manifest.json"
        self._load_manifest()

Registration and Deduplication

The register_file() method copies source files into the store (unless copy_file=False), generates a UUID-based attachment_id, and computes the SHA-256 hash. When deduplicate=True, the store checks _hash_index before writing; if an identical file exists, it returns the existing AttachmentRecord, avoiding redundant disk usage.

Manifest Persistence

Persistent attachments are recorded in attachments_manifest.json, which maps attachment_id to AttachmentRecord objects. The manifest includes the MIME type, original filename, size, hash, and local storage path, ensuring artifacts survive across workflow restarts. The store automatically synchronizes this JSON file via _save_manifest() after every registration.

Retrieval as Message Blocks

The to_message_block(attachment_id) method converts stored files into MessageBlock objects defined in entity/messages.py. Using the stored MIME type, it selects the appropriate MessageBlockType (e.g., image, text, or binary), preparing the artifact for direct inclusion in LLM prompts or UI rendering.

Workspace Scanner for UI Integration

For frontend components that need to browse workspace contents without loading file data, utils/workspace_scanner.py provides the iter_workspace_entries() generator.

# utils/workspace_scanner.py

def iter_workspace_entries(root, recursive=True, max_depth=5, include_hidden=False):
    # yields WorkspaceEntry objects (path, type, size, modified_ts, depth)

The scanner respects the max_depth parameter (default 5) to prevent stack overflow on deeply nested directories, and filters hidden files (those starting with .) unless include_hidden=True. Each yielded WorkspaceEntry contains sufficient metadata—path, file type, size, and modification timestamp—for the Vue-based editor to render directory trees and file icons efficiently.

Complete Implementation Workflow

The following example demonstrates initializing the attachment store, configuring the artifact hook, and integrating both with a workflow executor:

from workflow.hooks.workspace_artifact import WorkspaceArtifactHook
from utils.attachments import AttachmentStore
from utils.workspace_scanner import iter_workspace_entries

# 1. Initialize persistent storage (shared across workflow run)

attachment_store = AttachmentStore(root_dir="/tmp/chatdev_attachments")

# 2. Define callback to handle artifact changes

def on_artifacts(artifacts):
    for art in artifacts:
        print(f"[{art.change_type}] {art.relative_path} → {art.attachment_id}")

# 3. Configure hook for Python and agent nodes only

artifact_hook = WorkspaceArtifactHook(
    attachment_store=attachment_store,
    emit_callback=on_artifacts,
    node_types=["python", "agent"],
    exclude_dirs=["__pycache__", "temp", "attachments"],
)

# 4. Execute node with lifecycle hooks

def run_node(node, workspace_path):
    artifact_hook.before_node(node, workspace_path)
    success = node.run(workspace_path)  # Node writes output.png, logs, etc.

    artifact_hook.after_node(node, workspace_path, success=success)

# 5. UI browsing capability

for entry in iter_workspace_entries(workspace_path, include_hidden=False, max_depth=3):
    print(f"{entry.path} ({entry.type}): {entry.size} bytes")

When a node writes output.png to its workspace, the hook detects the change via SHA-256 comparison, the AttachmentStore persists it under a content-addressed UUID, and on_artifacts receives a WorkspaceArtifact that can be transformed into a multimodal message block for the LLM.

Summary

• WorkspaceArtifactHook snapshots workspaces before and after node execution, using SHA-256 hashes to detect incremental changes while respecting file-count and byte-size limits.
• AttachmentStore provides deduplicated, filesystem-backed persistence with a JSON manifest, converting files into LLM-ready MessageBlock objects via to_message_block().
• WorkspaceScanner enables UI-friendly directory traversal without loading file contents, supporting depth limits and hidden file filters.
• The architecture cleanly separates detection (workflow/hooks/workspace_artifact.py) from persistence (utils/attachments.py), allowing alternative backends (cloud storage, databases) to integrate without modifying the workflow runtime.

Frequently Asked Questions

How does ChatDev detect which files changed in a workspace?

The system uses the WorkspaceArtifactHook in workflow/hooks/workspace_artifact.py to take SHA-256 hash snapshots of the workspace immediately before and after node execution. By comparing these two states in after_node(), it identifies created, updated, or deleted files without requiring OS-specific file watchers or continuous polling.

What prevents duplicate files from consuming extra disk space?

The AttachmentStore implements content-addressed storage using SHA-256 hashes. When register_file() is called with deduplicate=True, the store checks _hash_index; if the hash exists, it returns the existing AttachmentRecord rather than copying the file, ensuring identical files across different nodes or runs reference a single physical copy.

How are workspace files presented to the LLM as multimodal inputs?

After registration, the to_message_block(attachment_id) method retrieves the AttachmentRecord and constructs a MessageBlock object from entity/messages.py. The store uses the recorded MIME type to determine the correct MessageBlockType (image, text, or binary), allowing the workflow to embed file references directly into LLM message sequences.

Can the UI browse workspace contents without downloading entire files?

Yes, the iter_workspace_entries() function in utils/workspace_scanner.py provides lightweight metadata scanning. It yields WorkspaceEntry objects containing paths, types, sizes, and timestamps without reading file contents, and supports max_depth and include_hidden parameters to optimize rendering for large directory trees.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}