How to Implement Custom Perception Detectors Utilizing VLM Models in DimOS

DimOS provides a VlModel abstraction in dimos/models/vl/base.py that wraps Vision-Language Models and exposes query_detections() and query_points() methods, allowing you to build custom perception capabilities by subclassing the base class or injecting existing implementations into skill-decorated modules.

The dimensionalOS/dimos repository treats Vision-Language Models (VLMs) as modular resources that power perception pipelines. By leveraging the VlModel base class and detection conversion helpers, you can implement custom perception detectors utilizing VLM models with minimal boilerplate while maintaining full integration with the agentic skill system.

Understanding the VLM Perception Architecture

DimOS centralizes VLM interactions through the VlModel base class defined in dimos/models/vl/base.py. This resource-oriented module provides lifecycle management, image preprocessing via _prepare_image(), and generic query APIs including query(), query_batch(), and query_multi().

The architecture relies on three core components:

• Detection Type System: Located in dimos/perception/detection/type/__init__.py, this defines Detection2DBBox for bounding boxes and Detection2DPoint for keypoint detection, both wrapped in an ImageDetections2D container.

• Conversion Helpers: The vlm_detection_to_detection2d() function (lines 57‑71 in dimos/models/vl/base.py) parses JSON responses like [label, x1, y1, x2, y2] into typed Detection2DBBox objects, while vlm_point_to_detection2d_point() handles point detections.

• Concrete Implementations: Model-specific wrappers such as MoondreamVlModel in dimos/models/vl/moondream.py and QwenVlModel in dimos/models/vl/qwen.py implement the low-level query() method while inheriting high-level detection capabilities from the base class.

When an agent requests object detection, the system calls query_detections() (lines 61‑75), which constructs a JSON-formatted prompt, executes the VLM query, and converts the raw response into standardized detection objects that integrate with downstream mapping and planning modules.

Building a Custom Detector with Existing VLMs

You can create specialized detectors by injecting an existing VLM implementation into a module and calling its detection methods. The following example uses Moondream to detect sports balls:

# detectors/custom_ball_detector.py

from dataclasses import dataclass
from dimos.agents.annotation import skill
from dimos.core.module import Module, ModuleConfig
from dimos.core.core import rpc
from dimos.msgs.sensor_msgs import Image
from dimos.models.vl.moondream import MoondreamVlModel
from dimos.perception.detection.type import ImageDetections2D

@dataclass
class BallDetectorConfig(ModuleConfig):
    max_objects: int = 5
    confidence_threshold: float = 0.8

class BallDetector(Module):
    """Detect spherical objects using Moondream VLM."""
    default_config = BallDetectorConfig
    config: BallDetectorConfig
    
    _vlm: MoondreamVlModel  # Injected by blueprint

    @rpc
    def start(self) -> None:
        super().start()

    @skill
    def detect_balls(self, image: Image) -> ImageDetections2D:
        """
        Detect balls in the provided image.
        
        Args:
            image: RGB image from a camera sensor.
        Returns:
            ImageDetections2D with bounding box detections.
        """
        detections = self._vlm.query_detections(
            image,
            query="ball",
            max_objects=self.config.max_objects,
        )
        
        # Filter by minimum width (20 pixels)

        filtered = [
            d for d in detections.detections 
            if (d.bbox[2] - d.bbox[0]) > 20
        ]
        detections.detections = filtered
        
        return detections

ball_detector = BallDetector.blueprint

Key implementation details:

• Dependency injection: The _vlm attribute is automatically populated by the DimOS blueprint system using autoconnect composition.
• Prompt engineering: The query_detections() method automatically wraps your string query (e.g., "ball") into a JSON prompt structure expected by the VLM.
• Native types: The returned ImageDetections2D object is compatible with the broader perception pipeline in dimos/perception/.

Supporting New VLM Providers

To integrate a proprietary or unsupported VLM, subclass VlModel and override the query() method. The base class handles the rest, including detection formatting via vlm_detection_to_detection2d().

# models/vl/custom_vlm.py

import json
import requests
from dataclasses import dataclass
from dimos.models.vl.base import VlModel, VlModelConfig
from dimos.msgs.sensor_msgs import Image

@dataclass
class CustomVLMConfig(VlModelConfig):
    api_key: str | None = None
    endpoint: str = "https://api.customvlm.com/v1/analyze"

class CustomVLM(VlModel):
    default_config = CustomVLMConfig
    config: CustomVLMConfig

    def query(self, image: Image, query: str, **kwargs: dict) -> str:
        """Execute low-level VLM query."""
        img_b64 = image.to_base64()
        payload = {
            "image": img_b64,
            "prompt": query,
        }
        headers = {"Authorization": f"Bearer {self.config.api_key}"}
        
        resp = requests.post(
            self.config.endpoint, 
            json=payload, 
            headers=headers, 
            timeout=30
        )
        resp.raise_for_status()
        
        return resp.json()["answer"]

Once registered, this implementation inherits query_detections() and query_points() automatically, allowing immediate use in perception skills without modifying the core framework.

Integrating with the Agent Skill System

Expose your detector to LLM agents by applying the @skill decorator from dimos.agents.annotation. This registers the method as a tool that agents can invoke through the MCP interface or direct function calling.

Reference implementations in dimos/perception/perceive_loop_skill.py demonstrate how to structure perception skills that stream images and query VLMs. When an agent receives a prompt like "Find red chairs in the current view", the system instantiates your detector module and executes the skill method, returning structured detection data that the agent can incorporate into its reasoning.

Because query_detections() returns standardized ImageDetections2D objects, agents can programmatically access pixel coordinates, bounding box dimensions, and detection labels without parsing raw VLM text outputs.

Summary

• DimOS abstracts VLM complexity through the VlModel base class in dimos/models/vl/base.py, providing query_detections() and query_points() methods that return typed detection objects.

• Existing implementations like MoondreamVlModel and QwenVlModel can be injected into custom modules via the blueprint system, requiring only prompt definition and optional post-processing.

• New VLM providers integrate by subclassing VlModel and implementing the query() method; detection formatting and type conversion are handled automatically by vlm_detection_to_detection2d().

• LLM accessibility is achieved through the @skill decorator, which exposes detector methods as tools available to agents via the MCP interface.

• Type safety is maintained throughout the pipeline using Detection2DBBox, Detection2DPoint, and ImageDetections2D from dimos/perception/detection/type/__init__.py.

Frequently Asked Questions

Can I use any Vision-Language Model with DimOS?

Yes. You can wrap any VLM that accepts image inputs and returns text or JSON by subclassing VlModel in dimos/models/vl/base.py and implementing the query() method. The base class handles prompt formatting through query_detections() and result parsing through vlm_detection_to_detection2d(), so you only need to manage the network call and authentication.

How do I filter detections by confidence or size?

Because query_detections() returns an ImageDetections2D object containing a list of Detection2DBBox instances, you can apply standard Python filtering before returning the result. Access the bounding box coordinates via the bbox attribute (format [x1, y1, x2, y2]) and filter by width, height, or area thresholds as shown in the custom ball detector example.

What is the difference between query_detections and query_points?

query_detections() (lines 61‑75 in dimos/models/vl/base.py) is optimized for bounding box outputs and uses vlm_detection_to_detection2d() to parse JSON arrays of [label, x1, y1, x2, y2]. query_points() uses vlm_point_to_detection2d_point() for single-coordinate outputs like [label, x, y], returning Detection2DPoint objects suitable for keypoint or landmark detection tasks.

How do agents discover and call custom detectors?

Agents discover detectors through the @skill decorator, which registers the method as a tool in the DimOS skill registry. When you define a detector method with this decorator in a module, it becomes available for LLM function calling and can be invoked via the MCP command-line interface using dimos mcp call <skill_name>, or automatically by agents equipped with the perception capability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}