# VSS Incident Formatting: How the Multi-Incident Formatter Workflow Works

> Understand VSS incident formatting. Learn how the NVIDIA multi_incident_formatter workflow normalizes timestamps and generates JSON payloads with video URLs and charts.

- Repository: [NVIDIA AI Blueprints/video-search-and-summarization](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization)
- Tags: how-to-guide
- Published: 2026-05-15

---

**The `multi_incident_formatter` tool in NVIDIA's Video Search & Summarization (VSS) platform fetches raw video analytics incidents, normalizes timestamps to ISO-8601 format, and produces a JSON payload with embedded video URLs and optional visualization charts.**

The **Video Search & Summarization (VSS)** platform processes raw incident data from video analytics pipelines and transforms it into user-facing formats. The `multi_incident_formatter` workflow, implemented in the `NVIDIA-AI-Blueprints/video-search-and-summarization` repository, provides a robust mechanism for retrieving, formatting, and visualizing security incidents across sensors and places. This tool bridges the gap between backend analytics and frontend presentation by handling timestamp normalization, media URL generation, and optional chart rendering.

## Multi-Incident Formatter Architecture

The formatter follows a declarative configuration pattern using Pydantic models to define inputs, outputs, and behavior. This architecture separates data fetching from presentation logic, enabling reuse across different VSS agent implementations.

### Configuration and Data Models

Three core Pydantic models govern the formatter's operation:

- **`MultiIncidentFormatterConfig`** – Defines which sub-tools to invoke (`video_url_tool`, `picture_url_tool`, `incidents_tool`, `chart_generator_tool`) and behavioral flags including `generate_chart` and `display_limit`. Located in [`agent/src/vss_agents/tools/multi_incident_formatter.py`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/main/agent/src/vss_agents/tools/multi_incident_formatter.py) at line 62.
- **`MultiIncidentFormatterInput`** – Validates user requests containing `source` identifiers, `source_type` (sensor or place), optional time windows, and a hard-capped `max_result_size`. Includes automatic timestamp normalization via Pydantic validators (line 108).
- **`MultiIncidentFormatterOutput`** – Returns a structured response containing the `<incidents>`-wrapped JSON string, total incident count, and optional chart HTML (line 142).

### Supporting Components

The formatter integrates with several specialized tools from the VSS toolchain:

- **[`video_url_tool.py`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/main/video_url_tool.py)** – Generates video clip URLs for specified time windows
- **[`picture_url_tool.py`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/main/picture_url_tool.py)** – Provides snapshot image URLs for individual incidents  
- **[`incidents_tool.py`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/main/incidents_tool.py)** – Retrieves raw incident data from the analytics backend
- **[`chart_generator.py`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/main/chart_generator.py)** – Creates visualization charts stored in object storage

## How the Formatter Workflow Executes

The implementation in [`agent/src/vss_agents/tools/multi_incident_formatter.py`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/main/agent/src/vss_agents/tools/multi_incident_formatter.py) orchestrates incident retrieval and formatting through a series of async helper functions.

### Incident Retrieval and Normalization

The **`_fetch_incidents`** helper (line 59) handles the initial data acquisition:

1. Invokes the configured *incidents* tool with the user's source and time window parameters
2. Parses JSON-string results and handles malformed entries through defensive error logging
3. Constructs a list of internal `IncidentData` objects with **strict ISO-8601 timestamp normalization** to millisecond precision

This ensures consistent temporal representation regardless of the upstream analytics pipeline's output format.

### Parallel Incident Formatting

Once incidents are retrieved, the **`_format_single_incident`** function (line 28) processes each event individually:

- Calls the *video URL* and *picture URL* sub-tools to generate media links
- Extracts verification data when present in the raw incident payload
- Assembles a standardized JSON fragment containing `Alert Title`, `Clip Information`, and `Alert Details`
- Implements error handling that returns fallback structures when media generation fails

The **`_multi_incident_formatter_impl`** orchestrator (line 46) manages concurrent processing using `asyncio.gather`, limiting concurrent operations to the `display_limit` parameter (default 20).

### Workflow Execution Steps

The complete execution flow proceeds as follows:

1. **Invocation** – The VSS orchestration layer calls the registered `multi_incident_formatter` function with a concrete `MultiIncidentFormatterConfig`
2. **Tool Resolution** – The builder retrieves concrete implementations for video, picture, and incidents tools
3. **Input Validation** – User input passes through `MultiIncidentFormatterInput` validation (e.g., `source="sensor-42"`, `source_type="sensor"`)
4. **Data Fetching** – `_fetch_incidents` contacts the incidents service and normalizes timestamps
5. **Parallel Processing** – Up to `display_limit` incidents are formatted concurrently via `_format_single_incident`
6. **Response Assembly** – Formatted incidents are wrapped in `<incidents>` tags and combined with the total count

## Chart Generation and Visualization

When `generate_chart=True` and a chart generator is configured, the formatter produces two complementary visualizations based on **all** fetched incidents, not just the displayed subset.

### Visualization Components

- **`_generate_incidents_chart`** (line 13) – Creates a pie chart showing the distribution of incident categories
- **`_generate_time_series_chart`** (line 50) – Generates a time-series bar chart showing incident frequency over time
- **`_determine_optimal_bin_size`** (line 61) – Automatically calculates appropriate time buckets based on incident density to maintain readability

The bin size algorithm prevents overcrowding in the time-series visualization by dynamically adjusting the granularity based on the temporal spread of the incident dataset.

### Chart Output

Generated charts return as HTML `<img>` elements referencing the object storage URLs, embedded directly in the `MultiIncidentFormatterOutput.chart_html` field for immediate frontend rendering.

## Implementation Example

### Direct Python Integration

You can invoke the formatter programmatically within a VSS agent using the builder pattern:

```python
from vss_agents.tools.multi_incident_formatter import (
    MultiIncidentFormatterConfig,
    MultiIncidentFormatterInput,
    multi_incident_formatter,
)

# Initialize with pre-configured tool references

config = MultiIncidentFormatterConfig(
    video_url_tool=FunctionRef(name="video_url_tool"),
    picture_url_tool=FunctionRef(name="picture_url_tool"),
    incidents_tool=FunctionRef(name="incidents_tool"),
    chart_generator_tool=FunctionRef(name="chart_generator_tool"),
    generate_chart=True,
    display_limit=20,
)

# Register and invoke the function

async for fn_info in multi_incident_formatter(config, builder):
    result = await fn_info.single_fn(
        MultiIncidentFormatterInput(
            source="sensor-42",
            source_type="sensor",
            start_time="2025-09-22T14:00:00.000Z",
            end_time="2025-09-22T15:00:00.000Z",
        )
    )
    print(result.formatted_incidents)  # JSON wrapped in <incidents> tags

    print(result.chart_html)           # Visualization HTML

```

### LLM Orchestrator Integration

The tool exposes a LangChain-compatible interface for prompt-driven invocation:

```json
{
  "tool": "multi_incident_formatter",
  "input": {
    "source": "MainLobby",
    "source_type": "place",
    "start_time": "2025-09-20T00:00:00.000Z",
    "end_time": "2025-09-21T23:59:59.999Z",
    "max_result_size": 5000
  }
}

```

The VSS orchestrator processes this JSON, invokes the registered function, and embeds the returned incidents and charts into the conversation context.

## Summary

- **`multi_incident_formatter`** transforms raw video analytics data into UI-ready JSON payloads with embedded media URLs
- The workflow enforces **ISO-8601 timestamp normalization** and processes incidents in parallel for performance
- **Declarative configuration** via `MultiIncidentFormatterConfig` allows flexible tool composition without code changes
- Optional **chart generation** produces pie charts (category distribution) and time-series bar charts (temporal patterns) using automatic bin sizing
- The implementation resides in [`agent/src/vss_agents/tools/multi_incident_formatter.py`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/main/agent/src/vss_agents/tools/multi_incident_formatter.py) with comprehensive unit tests in [`agent/tests/unit_test/tools/test_multi_incident_formatter.py`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/main/agent/tests/unit_test/tools/test_multi_incident_formatter.py)

## Frequently Asked Questions

### How does the multi_incident_formatter handle timestamp normalization?

The formatter enforces strict ISO-8601 formatting with millisecond precision through Pydantic validators in `MultiIncidentFormatterInput`. When `_fetch_incidents` retrieves raw data from the incidents service, it normalizes all temporal fields to ensure consistency across different analytics pipeline outputs, preventing format mismatches in the final JSON payload.

### What is the difference between `display_limit` and `max_result_size`?

`max_result_size` (specified in `MultiIncidentFormatterInput`) caps the total number of incidents fetched from the backend service, while `display_limit` (set in `MultiIncidentFormatterConfig`) controls how many incidents are actually formatted and returned in the JSON payload. Visualizations (charts) are generated using the full fetched dataset regardless of the `display_limit` setting.

### Can the formatter operate without chart generation capabilities?

Yes. Chart generation is optional and controlled by the `generate_chart` boolean in `MultiIncidentFormatterConfig`. If set to `False` or if the `chart_generator_tool` is not provided, the workflow skips the visualization steps and returns a `MultiIncidentFormatterOutput` with `chart_html` set to `None`, containing only the formatted incidents and total count.

### How does the formatter handle errors in individual incident processing?

The `_format_single_incident` function implements defensive error handling that catches exceptions during video/picture URL generation or data extraction. Rather than failing the entire batch, it returns a fallback JSON structure for that specific incident containing error information, allowing the workflow to complete successfully while logging individual failures for debugging.