How the Data Flow Renderer Visualizes Pipelines and Data Lineage in Archify

The Archify Data Flow renderer transforms JSON specifications into interactive SVG diagrams that map data movement through transformation stages, highlighting privacy-sensitive boundaries and processing paths with visual variants.

The Data Flow renderer is one of five built-in diagram types in the Archify open-source repository (tt-a1i/archify). It converts structured JSON descriptions into layered visualizations that make complex data pipelines, lineage relationships, and compliance boundaries immediately comprehensible.

Core Concepts of Data Flow Visualization

The renderer operates on a schema-validated JSON structure that defines five key elements:

Element	JSON Expression	Visual Treatment
Stages	Ordered list in stages array	Horizontal rows grouping nodes by processing layer
Nodes	Objects with id, label, type, stage	Boxes or circles positioned in stage rows; type determines icon and color
Flows	Edge objects with source, target, variant, optional via and labelAt	Arrows with style driven by variant
Legend	Boolean legend flag and cards array	Side panel explaining variants and summarizing paths
Annotations	label on flows, note on nodes	Inline text or hover tooltips

Flow Variants and Their Visual Meanings

The variant field is the primary mechanism for communicating data lineage characteristics:

• emphasis — Solid bold arrow for primary analytic paths
• security — Dashed rose line with lock icon for PII or privacy boundaries
• dashed — Simple dashed line for batch or derived processing

JSON Schema and Validation

All Data Flow specifications validate against archify/schemas/dataflow.schema.json. This schema enforces required fields and constrains variant values to ensure consistent rendering.

A valid Data Flow JSON requires three top-level arrays:

1. stages — Defines the horizontal layers (e.g., Sources → Ingest → Process → Store → Consume)
2. nodes — Places components into stages with type classification
3. flows — Connects nodes with directed edges carrying variant metadata

Rendering Pipeline: From JSON to SVG

The Data Flow renderer executes a four-stage transformation process:

1. Stage layout assigns fixed vertical slots; nodes distribute evenly across width
2. Edge routing draws paths from source to target, respecting optional via waypoints
3. Style application maps variant values to CSS classes controlling line style, color, and icons
4. Interactivity binding attaches hover states for labels/notes and click handlers for cards

Complete Example: Product Analytics Data Flow

The repository includes a fully-featured example at archify/examples/product-analytics.dataflow.json. Below is a simplified variant demonstrating all core features:

{
  "title": "Simple Order Processing Data Flow",
  "stages": [
    { "id": "sources", "label": "Sources" },
    { "id": "ingest", "label": "Ingest" },
    { "id": "process", "label": "Process" },
    { "id": "store", "label": "Store" },
    { "id": "consume", "label": "Consume" }
  ],
  "nodes": [
    { "id": "web-sdk", "label": "Web SDK", "stage": "sources" },
    { "id": "edge-api", "label": "Edge API", "stage": "ingest" },
    { "id": "consent", "label": "Consent Gate", "stage": "process", "type": "security" },
    { "id": "event-bus", "label": "Event Bus", "stage": "process" },
    { "id": "warehouse", "label": "Analytics Warehouse", "stage": "store" },
    { "id": "dashboard", "label": "Dashboard", "stage": "consume" }
  ],
  "flows": [
    { "source": "web-sdk", "target": "edge-api", "variant": "emphasis" },
    { "source": "edge-api", "target": "consent", "variant": "security" },
    { "source": "consent", "target": "event-bus", "variant": "security" },
    { "source": "event-bus", "target": "warehouse", "variant": "emphasis" },
    { "source": "warehouse", "target": "dashboard", "variant": "emphasis" }
  ],
  "legend": true,
  "cards": [
    {
      "title": "Primary Path",
      "variant": "emphasis",
      "summary": "Web → Edge → Event Bus → Warehouse → Dashboard"
    },
    {
      "title": "Privacy Boundary",
      "variant": "security",
      "summary": "Consent Gate protects PII"
    }
  ]
}

Save as simple-order.dataflow.json and render with:

archify view simple-order.dataflow.json

Viewing the Live Demo

The repository provides a pre-rendered HTML demonstration at examples/dataflow-product-analytics.html. To launch it locally:

# After cloning tt-a1i/archify

archify view https://github.com/tt-a1i/archify/blob/main/archify/examples/product-analytics.dataflow.json

This renders five stages, ten nodes, and ten variant-coded flows with interactive legend and summary cards.

Key Source Files

Path	Purpose
archify/schemas/dataflow.schema.json	Validation schema for Data Flow JSON
archify/renderers/dataflow/README.md	Renderer configuration and embedding guide
archify/examples/product-analytics.dataflow.json	Full-featured example specification
examples/dataflow-product-analytics.html	Browser-rendered output demonstration
docs/assets/archify-dataflow.png	Visual summary for documentation

What Data Lineage Visualization Reveals

The Archify Data Flow renderer makes three critical aspects of data architecture explicit:

• End-to-end pipelines — Complete traceability from ingestion through consumption
• Privacy boundaries — Immediate visual identification of PII touchpoints via security variant styling
• Processing modes — Distinction between real-time and batch paths through emphasis vs dashed variants

Summary

• The Data Flow renderer converts JSON to interactive SVG using a validated schema-based pipeline
• Flow variants (emphasis, security, dashed) encode semantic meaning about data lineage and sensitivity
• Stages create horizontal layers that correspond to logical processing phases
• Cards and legends provide contextual summaries alongside the diagram
• All implementations reference source files in archify/schemas/, archify/renderers/dataflow/, and archify/examples/

Frequently Asked Questions

What JSON schema does the Data Flow renderer use?

The renderer validates input against archify/schemas/dataflow.schema.json, which defines required stages, nodes, and flows arrays plus constrained variant values. Validation ensures consistent rendering across all Data Flow diagrams.

How do I mark sensitive data flows in a Data Flow diagram?

Apply the security variant to flows handling PII or crossing privacy boundaries. This renders as a dashed rose line with a lock icon, making sensitive paths immediately visible in the visualization.

Can I customize node placement within stages?

Yes. Use the optional via array in flow definitions to specify waypoint coordinates. The renderer routes edges through these points, enabling precise control over path geometry and overlap avoidance.

Where can I see a complete rendered example?

The file examples/dataflow-product-analytics.html in the tt-a1i/archify repository contains a live demonstration. Open it in any browser after cloning, or use archify view with the remote URL to render the bundled product-analytics.dataflow.json example.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}