How Archify Visualizes Data Pipelines with the Dataflow Renderer

Archify converts declarative JSON pipeline descriptions into polished, interactive SVG diagrams through a six-stage layout, routing, and rendering pipeline driven by render-dataflow.mjs.

The dataflow renderer is the core visualization engine in tt-a1i/archify, transforming abstract data pipeline definitions into publication-ready diagrams. It treats every pipeline as a data-flow diagram composed of stages (columns), nodes (processing units), and flows (data movement arrows). This article explains how the renderer works, from JSON input to SVG output, with specific references to the source implementation.

Loading the Pipeline Definition

The renderer begins by ingesting a JSON intermediate representation (IR) that declares the complete pipeline structure. The entry point loadDiagramWithBrandMarks in archify/renderers/dataflow/render-dataflow.mjs parses this IR, extracting stages, nodes, flows, and optional metadata including view-box dimensions and animation profiles【1†L45-L49】.

The default example product-analytics.dataflow.json demonstrates a complete IR with business domains, security boundaries, and narrative cards【2†L1-L46】:

{
  "diagram_type": "dataflow",
  "stages": [
    { "label": "Source" },
    { "label": "Process" },
    { "label": "Sink" }
  ],
  "nodes": [
    { "id": "api", "type": "frontend", "label": "API Gateway", "stage": 0, "row": 0 },
    { "id": "worker", "type": "backend", "label": "Worker Pool", "stage": 1, "row": 0 },
    { "id": "db", "type": "database", "label": "Analytics DB", "stage": 2, "row": 0 }
  ],
  "flows": [
    { "from": "api", "to": "worker", "label": "events" },
    { "from": "worker", "to": "db", "label": "aggregations" }
  ]
}

Calculating the Layout

The layout engine computes precise geometry for every visual element before generating SVG.

Stage Positioning

Stages receive horizontal placement through stageX, stageW, and colGap parameters. Each stage renders as a rectangular stageFrame that visually groups its contained nodes【1†L73-L88】.

Node Measurement

The measureNode function determines each node's width, height, center-point (cx), and vertical position (y) based on its assigned stage and row indices. This ensures consistent spacing regardless of label length or node type【1†L92-L105】.

Flow Routing

Flows undergo validation for unique IDs, valid node references, minimum path length, and non-diagonal routing constraints. The routeVia function applies explicit routing directives when provided via via, channelX, channelY, or route properties; otherwise, automatic orthogonal routing selects the shortest valid path【1†L96-L119】【1†L124-L142】.

Generating Connection Paths

For each validated flow, the renderer:

1. Obtains start and end attachment points via the anchor function
2. Applies routing hints through routeVia
3. Removes degenerate or redundant path points
4. Constructs an SVG polyline stored in pathCache for reuse【1†L37-L63】

This caching strategy prevents redundant path calculations when the same connection geometry appears multiple times or requires label positioning.

Rendering to SVG

The rendering phase executes four specialized functions that emit SVG elements:

• renderStage — draws stage frames as <rect> elements with labels【1†L65-L70】
• renderNode — draws nodes as styled rectangles with type-based fills, optional sub-labels, tags, and brand marks【1†L72-L95】
• renderFlowPath — draws flows as <path> elements with computed points and arrowhead markers【1†L98-L104】
• renderFlowLabel — positions descriptive labels at the midpoint of each flow path【1†L105-L115】

The renderer also generates an automatic legend via renderLegend, mapping flow variants (e.g., "emphasis", "dashed", "secure") to visual styles【1†L118-L132】.

Validation and Diagnostics

Before final output, validateDataflow() traverses all nodes and flows to detect:

• Overlapping node geometries
• Out-of-bounds coordinates
• Flow label collisions
• Disconnected or invalid references

Detected problems accumulate in a diagnostic structure; if any exist, throwDiagnosticProblems halts execution with a detailed report【1†L18-L93】. This strict validation ensures output quality and catches subtle layout errors early.

Writing Final Output

The completed SVG includes:

• Accessibility metadata and <title> elements
• <defs> for gradients, patterns, and reusable arrowheads
• Optional background grid
• Optional narrative cards for annotations
• Responsive viewBox sizing

writeDiagram serializes this structure to the target file path【1†L74-L84】.

Command-Line and Programmatic Usage

Generate diagrams directly from the CLI:

npx archify --type dataflow --source examples/product-analytics.dataflow.json

Output writes to examples/dataflow-product-analytics.html with embedded SVG and interactive controls.

Embed finished diagrams in documentation:

<iframe src="gallery/artifacts/product-analytics.dataflow.html?embed=1&theme=dark"
        title="Product Analytics data flow"
        width="100%"
        height="600"
        loading="lazy">
</iframe>

Key Implementation Files

File	Purpose
archify/renderers/dataflow/render-dataflow.mjs	Core renderer implementing layout, validation, and SVG generation【1†L45-L84】【1†L92-L130】
examples/product-analytics.dataflow.json	Reference IR with stages, flows, security zones, and cards【2†L1-L46】
docs/gallery.html	Gallery loader for rendered artifacts【1†L428-L452】
scripts/start-template.html	CLI bootstrap template for new dataflow diagrams【1†L221-L284】

Summary

• Input format: Declarative JSON IR with stages, nodes, and flows
• Layout engine: Calculates geometry for stages, measures nodes, routes flows orthogonally
• Path generation: Caches polylines with anchor points and routing hints
• Rendering: Specialized functions emit stage frames, node boxes, flow paths, labels, and legends
• Quality assurance: validateDataflow() enforces geometric and topological constraints
• Output: Optimized SVG with accessibility features, responsive sizing, and optional interactivity

Frequently Asked Questions

What file format does the dataflow renderer accept?

The renderer accepts a JSON intermediate representation with top-level keys for diagram_type (must be "dataflow"), stages (array of column definitions), nodes (array of processing units with stage/row positioning), and flows (array of connections with from/to node references). Optional keys include viewBox, theme, cards for annotations, and animation for temporal sequence rendering.

How does Archify prevent overlapping nodes or crossed flows?

The validateDataflow() function executes geometric checks before SVG generation. It computes axis-aligned bounding boxes for all nodes and flags intersections. For flows, it validates that automatic routing produces non-degenerate paths and that explicit via coordinates don't create self-intersections. Validation failures throw detailed diagnostics with node/flow IDs and specific problem descriptions【1†L18-L93】.

Can I customize the visual style of dataflow diagrams?

Yes. The IR accepts theme overrides for colors, variant properties on flows to select from predefined styles (solid, dashed, emphasis, secure), and brandMarks for logo overlays. The renderNode function selects fill colors based on node type values like "frontend", "backend", "database", or "queue"【1†L72-L95】. For deeper customization, modify the template in scripts/start-template.html or post-process the generated SVG.

Is the dataflow renderer suitable for real-time or animated pipelines?

The renderer supports static SVG output with optional animation sequences. The animation property in the IR can specify frame-by-frame visibility changes or staged reveals. For fully interactive or real-time visualizations, export the SVG and manipulate DOM elements via JavaScript, or regenerate the diagram when pipeline topology changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}