VSS Incident Formatting: How the Multi-Incident Formatter Workflow Works

The multi_incident_formatter tool in NVIDIA's Video Search & Summarization (VSS) platform fetches raw video analytics incidents, normalizes timestamps to ISO-8601 format, and produces a JSON payload with embedded video URLs and optional visualization charts.

The Video Search & Summarization (VSS) platform processes raw incident data from video analytics pipelines and transforms it into user-facing formats. The multi_incident_formatter workflow, implemented in the NVIDIA-AI-Blueprints/video-search-and-summarization repository, provides a robust mechanism for retrieving, formatting, and visualizing security incidents across sensors and places. This tool bridges the gap between backend analytics and frontend presentation by handling timestamp normalization, media URL generation, and optional chart rendering.

Multi-Incident Formatter Architecture

The formatter follows a declarative configuration pattern using Pydantic models to define inputs, outputs, and behavior. This architecture separates data fetching from presentation logic, enabling reuse across different VSS agent implementations.

Configuration and Data Models

Three core Pydantic models govern the formatter's operation:

• MultiIncidentFormatterConfig – Defines which sub-tools to invoke (video_url_tool, picture_url_tool, incidents_tool, chart_generator_tool) and behavioral flags including generate_chart and display_limit. Located in agent/src/vss_agents/tools/multi_incident_formatter.py at line 62.
• MultiIncidentFormatterInput – Validates user requests containing source identifiers, source_type (sensor or place), optional time windows, and a hard-capped max_result_size. Includes automatic timestamp normalization via Pydantic validators (line 108).
• MultiIncidentFormatterOutput – Returns a structured response containing the <incidents>-wrapped JSON string, total incident count, and optional chart HTML (line 142).

Supporting Components

The formatter integrates with several specialized tools from the VSS toolchain:

• video_url_tool.py – Generates video clip URLs for specified time windows
• picture_url_tool.py – Provides snapshot image URLs for individual incidents
• incidents_tool.py – Retrieves raw incident data from the analytics backend
• chart_generator.py – Creates visualization charts stored in object storage

How the Formatter Workflow Executes

The implementation in agent/src/vss_agents/tools/multi_incident_formatter.py orchestrates incident retrieval and formatting through a series of async helper functions.

Incident Retrieval and Normalization

The _fetch_incidents helper (line 59) handles the initial data acquisition:

1. Invokes the configured incidents tool with the user's source and time window parameters
2. Parses JSON-string results and handles malformed entries through defensive error logging
3. Constructs a list of internal IncidentData objects with strict ISO-8601 timestamp normalization to millisecond precision

This ensures consistent temporal representation regardless of the upstream analytics pipeline's output format.

Parallel Incident Formatting

Once incidents are retrieved, the _format_single_incident function (line 28) processes each event individually:

• Calls the video URL and picture URL sub-tools to generate media links
• Extracts verification data when present in the raw incident payload
• Assembles a standardized JSON fragment containing Alert Title, Clip Information, and Alert Details
• Implements error handling that returns fallback structures when media generation fails

The _multi_incident_formatter_impl orchestrator (line 46) manages concurrent processing using asyncio.gather, limiting concurrent operations to the display_limit parameter (default 20).

Workflow Execution Steps

The complete execution flow proceeds as follows:

1. Invocation – The VSS orchestration layer calls the registered multi_incident_formatter function with a concrete MultiIncidentFormatterConfig
2. Tool Resolution – The builder retrieves concrete implementations for video, picture, and incidents tools
3. Input Validation – User input passes through MultiIncidentFormatterInput validation (e.g., source="sensor-42", source_type="sensor")
4. Data Fetching – _fetch_incidents contacts the incidents service and normalizes timestamps
5. Parallel Processing – Up to display_limit incidents are formatted concurrently via _format_single_incident
6. Response Assembly – Formatted incidents are wrapped in <incidents> tags and combined with the total count

Chart Generation and Visualization

When generate_chart=True and a chart generator is configured, the formatter produces two complementary visualizations based on all fetched incidents, not just the displayed subset.

Visualization Components

• _generate_incidents_chart (line 13) – Creates a pie chart showing the distribution of incident categories
• _generate_time_series_chart (line 50) – Generates a time-series bar chart showing incident frequency over time
• _determine_optimal_bin_size (line 61) – Automatically calculates appropriate time buckets based on incident density to maintain readability

The bin size algorithm prevents overcrowding in the time-series visualization by dynamically adjusting the granularity based on the temporal spread of the incident dataset.

Chart Output

Generated charts return as HTML <img> elements referencing the object storage URLs, embedded directly in the MultiIncidentFormatterOutput.chart_html field for immediate frontend rendering.

Implementation Example

Direct Python Integration

You can invoke the formatter programmatically within a VSS agent using the builder pattern:

from vss_agents.tools.multi_incident_formatter import (
    MultiIncidentFormatterConfig,
    MultiIncidentFormatterInput,
    multi_incident_formatter,
)

# Initialize with pre-configured tool references

config = MultiIncidentFormatterConfig(
    video_url_tool=FunctionRef(name="video_url_tool"),
    picture_url_tool=FunctionRef(name="picture_url_tool"),
    incidents_tool=FunctionRef(name="incidents_tool"),
    chart_generator_tool=FunctionRef(name="chart_generator_tool"),
    generate_chart=True,
    display_limit=20,
)

# Register and invoke the function

async for fn_info in multi_incident_formatter(config, builder):
    result = await fn_info.single_fn(
        MultiIncidentFormatterInput(
            source="sensor-42",
            source_type="sensor",
            start_time="2025-09-22T14:00:00.000Z",
            end_time="2025-09-22T15:00:00.000Z",
        )
    )
    print(result.formatted_incidents)  # JSON wrapped in <incidents> tags

    print(result.chart_html)           # Visualization HTML

LLM Orchestrator Integration

The tool exposes a LangChain-compatible interface for prompt-driven invocation:

{
  "tool": "multi_incident_formatter",
  "input": {
    "source": "MainLobby",
    "source_type": "place",
    "start_time": "2025-09-20T00:00:00.000Z",
    "end_time": "2025-09-21T23:59:59.999Z",
    "max_result_size": 5000
  }
}

The VSS orchestrator processes this JSON, invokes the registered function, and embeds the returned incidents and charts into the conversation context.

Summary

• multi_incident_formatter transforms raw video analytics data into UI-ready JSON payloads with embedded media URLs
• The workflow enforces ISO-8601 timestamp normalization and processes incidents in parallel for performance
• Declarative configuration via MultiIncidentFormatterConfig allows flexible tool composition without code changes
• Optional chart generation produces pie charts (category distribution) and time-series bar charts (temporal patterns) using automatic bin sizing
• The implementation resides in agent/src/vss_agents/tools/multi_incident_formatter.py with comprehensive unit tests in agent/tests/unit_test/tools/test_multi_incident_formatter.py

Frequently Asked Questions

How does the multi_incident_formatter handle timestamp normalization?

The formatter enforces strict ISO-8601 formatting with millisecond precision through Pydantic validators in MultiIncidentFormatterInput. When _fetch_incidents retrieves raw data from the incidents service, it normalizes all temporal fields to ensure consistency across different analytics pipeline outputs, preventing format mismatches in the final JSON payload.

What is the difference between display_limit and max_result_size?

max_result_size (specified in MultiIncidentFormatterInput) caps the total number of incidents fetched from the backend service, while display_limit (set in MultiIncidentFormatterConfig) controls how many incidents are actually formatted and returned in the JSON payload. Visualizations (charts) are generated using the full fetched dataset regardless of the display_limit setting.

Can the formatter operate without chart generation capabilities?

Yes. Chart generation is optional and controlled by the generate_chart boolean in MultiIncidentFormatterConfig. If set to False or if the chart_generator_tool is not provided, the workflow skips the visualization steps and returns a MultiIncidentFormatterOutput with chart_html set to None, containing only the formatted incidents and total count.

How does the formatter handle errors in individual incident processing?

The _format_single_incident function implements defensive error handling that catches exceptions during video/picture URL generation or data extraction. Rather than failing the entire batch, it returns a fallback JSON structure for that specific incident containing error information, allowing the workflow to complete successfully while logging individual failures for debugging.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}