How the ChartGenerator Creates Visualizations from Video Analytics Data in NVIDIA VSS

The ChartGenerator tool converts structured JSON analytics results—such as object counts or activity distributions—into bar and pie charts using Matplotlib, encodes them as PNG or SVG images, uploads them to an object store, and returns HTML image tags for direct rendering in the chat UI.

The chart_generator is a core component of the Video Search & Summarization (VSS) agent package in the NVIDIA-AI-Blueprints/video-search-and-summarization repository. It serves as the bridge between raw numeric video analytics and visual insights that users can see in the conversational interface. By accepting standardized data structures and handling the entire rendering pipeline, the tool enables agents to visualize everything from hourly vehicle counts to activity label distributions without requiring custom plotting code.

Input Validation and Data Structures

The generation process begins with strict input validation defined in agent/src/vss_agents/tools/chart_generator.py. The tool expects a JSON payload that conforms to the ChartGeneratorInput model, which parses the request and prepares it for processing.

The input model contains:

• A list of charts_data items, where each item is either a BarChartData or PieChartData instance
• Optional output_dir and file_prefix fields for file naming

The output_dir validator (lines 107–122) strips any leading forward slash and resolves the path to an absolute string, ensuring clean URL construction later in the pipeline.

BarChartData (lines 62–72) captures grouped bar chart specifications:

• x_categories: List of category labels for the X-axis
• series: Dictionary mapping series names to lists of numeric values
• Optional axis labels and chart title

PieChartData (lines 62–72) handles proportional data:

• sizes: List of numeric values representing slice sizes
• labels: List of strings describing each slice
• Chart title

Both data classes inherit from ChartData, which defines the chart_file_format attribute supporting PNG, SVG, or JPEG outputs.

Rendering Charts with Matplotlib

Once validated, each entry in charts_data is routed to a type-specific plotting function in agent/src/vss_agents/tools/chart_generator.py.

Bar chart generation uses plot_bar_chart (lines 33–71):

1. Creates a figure and axis using plt.subplots()
2. Iterates through the series dictionary, calling ax.bar() for each data series to create grouped bars
3. Applies axis labels, titles, and legends
4. Adjusts layout with plt.tight_layout() to prevent label clipping

Pie chart generation uses plot_pie_chart (lines 74–88):

1. Generates the pie using ax.pie(sizes, labels=labels)
2. Optionally adds a legend for clarity
3. Applies the chart title

Both functions return a matplotlib.figure.Figure object ready for encoding.

Encoding and Object Storage

After rendering, the figure moves through the encoding and optional storage pipeline.

Format conversion is handled by convert_to_format (lines 110–115). This function writes the Matplotlib figure into an in-memory BytesIO buffer using the requested format (accessed via chart_file_format.value) and returns the raw image bytes.

Object-store upload occurs when the agent configuration provides an object_store_name and the user specifies an output_dir. The raw bytes are wrapped in an ObjectStoreItem with the appropriate content_type and uploaded via await object_store.upsert_object. The storage key follows the pattern <output_dir>/<file_prefix><index>.<ext>, ensuring organized retrieval.

HTML Delivery and UI Integration

The final step converts stored assets into renderable HTML. The _output_converter helper (lines 36–44) constructs an HTML snippet for each successful chart:

<img src="{base_url}{object_store_key}" alt="Image" />

The base_url is sourced from ChartGeneratorConfig.object_store_base_url, defaulting to http://localhost:8000/static/. This HTML string is returned to the chat UI, which automatically renders the visualizations alongside text responses.

Framework Registration

The tool integrates with the VSS agent through the Nat LLM framework using the @register_function decorator (lines 128–141). This registration supplies:

• Input and output schemas for type safety
• Conversion helpers for string-based and chat-based invocations
• Metadata required for the agent's function-calling router

Once registered, the agent can invoke chart_generator naturally during conversations, passing analytics results directly from video processing pipelines.

Practical Example: Visualizing Vehicle Counts

Below is a complete example demonstrating how to generate charts from video analytics data programmatically:

from vss_agents.tools.chart_generator import (
    ChartGeneratorConfig,
    ChartGeneratorInput,
    BarChartData,
    PieChartData,
    ChartFileFormat,
    chart_generator,
)
from nat.builder.builder import Builder

# Bar chart: Vehicle counts per time interval

bar_data = BarChartData(
    x_categories=["00:00", "01:00", "02:00"],
    series={"Cars": [12, 15, 9], "Bikes": [5, 3, 4]},
    title="Vehicle Count per Minute",
    x_label="Time",
    y_label="Count",
    chart_file_format=ChartFileFormat.PNG,
)

# Pie chart: Activity distribution

pie_data = PieChartData(
    sizes=[40, 30, 30],
    labels=["Walking", "Running", "Standing"],
    title="Activity Distribution",
    chart_file_format=ChartFileFormat.SVG,
)

input_data = ChartGeneratorInput(
    charts_data=[bar_data, pie_data],
    output_dir="charts",
    file_prefix="video_analytics_",
)

config = ChartGeneratorConfig(
    object_store_name="my_object_store",
    object_store_base_url="http://my-host/static/",
)

# Execute the generation

builder = Builder()
async for fn_info in chart_generator(config, builder):
    result = await fn_info.single_fn(input_data)
    print(result)  # Returns HTML img tags

When invoked via JSON from a VSS skill, the payload follows this structure:

{
  "charts_data": [
    {
      "x_categories": ["00:00", "01:00", "02:00"],
      "series": {"Cars": [12, 15, 9], "Bikes": [5, 3, 4]},
      "title": "Vehicle Count per Minute",
      "chart_file_format": "png"
    },
    {
      "sizes": [40, 30, 30],
      "labels": ["Walking", "Running", "Standing"],
      "title": "Activity Distribution",
      "chart_file_format": "svg"
    }
  ],
  "output_dir": "analytics_charts",
  "file_prefix": "demo_"
}

The tool returns HTML ready for embedding:

<img src="http://my-host/static/analytics_charts/demo_0.png" alt="Image" />
<img src="http://my-host/static/analytics_charts/demo_1.svg" alt="Image" />

Summary

• ChartGenerator in agent/src/vss_agents/tools/chart_generator.py provides end-to-end visualization for video analytics data.
• It accepts structured inputs via ChartGeneratorInput, supporting both BarChartData for categorical comparisons and PieChartData for proportional data.
• Matplotlib handles the rendering through plot_bar_chart and plot_pie_chart, producing publication-quality figures.
• The convert_to_format function encodes charts as PNG, SVG, or JPEG bytes for flexible delivery.
• Optional object-store integration enables persistent storage with automatic URL generation, while the _output_converter produces HTML image tags for immediate UI rendering.
• Registration via @register_function makes the tool available to the VSS agent's function-calling framework.

Frequently Asked Questions

What chart types does the ChartGenerator support?

The ChartGenerator supports bar charts and pie charts natively. Bar charts are ideal for time-series data like vehicle counts per frame or hourly activity summaries, while pie charts visualize proportional distributions such as the percentage of detected objects belonging to specific classes. Both types are defined in agent/src/vss_agents/tools/chart_generator.py using the BarChartData and PieChartData classes.

How does ChartGenerator handle image storage and retrieval?

If the VSS agent configuration includes an object_store_name and the request specifies an output_dir, the tool uploads rendered images to the configured object store using object_store.upsert_object. The generated key combines the output directory, file prefix, and index (e.g., charts/prefix_0.png). The tool then constructs HTML image tags using the object_store_base_url from ChartGeneratorConfig, allowing the chat UI to retrieve and display images via standard HTTP requests.

Can I customize the output format and resolution?

Yes. Each chart data structure inherits a chart_file_format attribute from the base ChartData class, accepting png, svg, or jpeg values. The convert_to_format function handles the encoding accordingly. While the current implementation uses Matplotlib defaults for resolution, the SVG format provides scalable vector graphics suitable for high-DPI displays without file size penalties.

How is ChartGenerator invoked during video analysis workflows?

The tool is registered with the Nat LLM framework using the @register_function decorator in agent/src/vss_agents/tools/chart_generator.py. During a conversation, when the agent determines that video analytics results should be visualized, it constructs a ChartGeneratorInput payload and invokes the registered function. The agent receives HTML strings in return and injects them into the chat response, causing the charts to render automatically in the user interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}