# How to Implement Custom Perception Detectors Utilizing VLM Models in DimOS

> Learn to implement custom perception detectors using VLM models in DimOS. Leverage the VlModel abstraction to build powerful custom perception capabilities with query detections and points.

- Repository: [Dimensional/dimos](https://github.com/dimensionalOS/dimos)
- Tags: how-to-guide
- Published: 2026-03-15

---

**DimOS provides a `VlModel` abstraction in [`dimos/models/vl/base.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/models/vl/base.py) that wraps Vision-Language Models and exposes `query_detections()` and `query_points()` methods, allowing you to build custom perception capabilities by subclassing the base class or injecting existing implementations into skill-decorated modules.**

The **dimensionalOS/dimos** repository treats Vision-Language Models (VLMs) as modular resources that power perception pipelines. By leveraging the `VlModel` base class and detection conversion helpers, you can implement custom perception detectors utilizing VLM models with minimal boilerplate while maintaining full integration with the agentic skill system.

## Understanding the VLM Perception Architecture

DimOS centralizes VLM interactions through the **`VlModel`** base class defined in [`dimos/models/vl/base.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/models/vl/base.py). This resource-oriented module provides lifecycle management, image preprocessing via `_prepare_image()`, and generic query APIs including `query()`, `query_batch()`, and `query_multi()`.

The architecture relies on three core components:

- **Detection Type System**: Located in [`dimos/perception/detection/type/__init__.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/perception/detection/type/__init__.py), this defines **`Detection2DBBox`** for bounding boxes and **`Detection2DPoint`** for keypoint detection, both wrapped in an **`ImageDetections2D`** container.

- **Conversion Helpers**: The `vlm_detection_to_detection2d()` function (lines 57‑71 in [`dimos/models/vl/base.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/models/vl/base.py)) parses JSON responses like `[label, x1, y1, x2, y2]` into typed `Detection2DBBox` objects, while `vlm_point_to_detection2d_point()` handles point detections.

- **Concrete Implementations**: Model-specific wrappers such as **`MoondreamVlModel`** in [`dimos/models/vl/moondream.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/models/vl/moondream.py) and **`QwenVlModel`** in [`dimos/models/vl/qwen.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/models/vl/qwen.py) implement the low-level `query()` method while inheriting high-level detection capabilities from the base class.

When an agent requests object detection, the system calls `query_detections()` (lines 61‑75), which constructs a JSON-formatted prompt, executes the VLM query, and converts the raw response into standardized detection objects that integrate with downstream mapping and planning modules.

## Building a Custom Detector with Existing VLMs

You can create specialized detectors by injecting an existing VLM implementation into a module and calling its detection methods. The following example uses **Moondream** to detect sports balls:

```python

# detectors/custom_ball_detector.py

from dataclasses import dataclass
from dimos.agents.annotation import skill
from dimos.core.module import Module, ModuleConfig
from dimos.core.core import rpc
from dimos.msgs.sensor_msgs import Image
from dimos.models.vl.moondream import MoondreamVlModel
from dimos.perception.detection.type import ImageDetections2D

@dataclass
class BallDetectorConfig(ModuleConfig):
    max_objects: int = 5
    confidence_threshold: float = 0.8

class BallDetector(Module):
    """Detect spherical objects using Moondream VLM."""
    default_config = BallDetectorConfig
    config: BallDetectorConfig
    
    _vlm: MoondreamVlModel  # Injected by blueprint

    @rpc
    def start(self) -> None:
        super().start()

    @skill
    def detect_balls(self, image: Image) -> ImageDetections2D:
        """
        Detect balls in the provided image.
        
        Args:
            image: RGB image from a camera sensor.
        Returns:
            ImageDetections2D with bounding box detections.
        """
        detections = self._vlm.query_detections(
            image,
            query="ball",
            max_objects=self.config.max_objects,
        )
        
        # Filter by minimum width (20 pixels)

        filtered = [
            d for d in detections.detections 
            if (d.bbox[2] - d.bbox[0]) > 20
        ]
        detections.detections = filtered
        
        return detections

ball_detector = BallDetector.blueprint

```

**Key implementation details:**

- **Dependency injection**: The `_vlm` attribute is automatically populated by the DimOS blueprint system using `autoconnect` composition.
- **Prompt engineering**: The `query_detections()` method automatically wraps your string query (e.g., `"ball"`) into a JSON prompt structure expected by the VLM.
- **Native types**: The returned `ImageDetections2D` object is compatible with the broader perception pipeline in `dimos/perception/`.

## Supporting New VLM Providers

To integrate a proprietary or unsupported VLM, subclass **`VlModel`** and override the `query()` method. The base class handles the rest, including detection formatting via `vlm_detection_to_detection2d()`.

```python

# models/vl/custom_vlm.py

import json
import requests
from dataclasses import dataclass
from dimos.models.vl.base import VlModel, VlModelConfig
from dimos.msgs.sensor_msgs import Image

@dataclass
class CustomVLMConfig(VlModelConfig):
    api_key: str | None = None
    endpoint: str = "https://api.customvlm.com/v1/analyze"

class CustomVLM(VlModel):
    default_config = CustomVLMConfig
    config: CustomVLMConfig

    def query(self, image: Image, query: str, **kwargs: dict) -> str:
        """Execute low-level VLM query."""
        img_b64 = image.to_base64()
        payload = {
            "image": img_b64,
            "prompt": query,
        }
        headers = {"Authorization": f"Bearer {self.config.api_key}"}
        
        resp = requests.post(
            self.config.endpoint, 
            json=payload, 
            headers=headers, 
            timeout=30
        )
        resp.raise_for_status()
        
        return resp.json()["answer"]

```

Once registered, this implementation inherits `query_detections()` and `query_points()` automatically, allowing immediate use in perception skills without modifying the core framework.

## Integrating with the Agent Skill System

Expose your detector to LLM agents by applying the **`@skill`** decorator from `dimos.agents.annotation`. This registers the method as a tool that agents can invoke through the MCP interface or direct function calling.

Reference implementations in [`dimos/perception/perceive_loop_skill.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/perception/perceive_loop_skill.py) demonstrate how to structure perception skills that stream images and query VLMs. When an agent receives a prompt like *"Find red chairs in the current view"*, the system instantiates your detector module and executes the skill method, returning structured detection data that the agent can incorporate into its reasoning.

Because `query_detections()` returns standardized `ImageDetections2D` objects, agents can programmatically access pixel coordinates, bounding box dimensions, and detection labels without parsing raw VLM text outputs.

## Summary

- **DimOS abstracts VLM complexity** through the `VlModel` base class in [`dimos/models/vl/base.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/models/vl/base.py), providing `query_detections()` and `query_points()` methods that return typed detection objects.

- **Existing implementations** like `MoondreamVlModel` and `QwenVlModel` can be injected into custom modules via the blueprint system, requiring only prompt definition and optional post-processing.

- **New VLM providers** integrate by subclassing `VlModel` and implementing the `query()` method; detection formatting and type conversion are handled automatically by `vlm_detection_to_detection2d()`.

- **LLM accessibility** is achieved through the `@skill` decorator, which exposes detector methods as tools available to agents via the MCP interface.

- **Type safety** is maintained throughout the pipeline using `Detection2DBBox`, `Detection2DPoint`, and `ImageDetections2D` from [`dimos/perception/detection/type/__init__.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/perception/detection/type/__init__.py).

## Frequently Asked Questions

### Can I use any Vision-Language Model with DimOS?

Yes. You can wrap any VLM that accepts image inputs and returns text or JSON by subclassing `VlModel` in [`dimos/models/vl/base.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/models/vl/base.py) and implementing the `query()` method. The base class handles prompt formatting through `query_detections()` and result parsing through `vlm_detection_to_detection2d()`, so you only need to manage the network call and authentication.

### How do I filter detections by confidence or size?

Because `query_detections()` returns an `ImageDetections2D` object containing a list of `Detection2DBBox` instances, you can apply standard Python filtering before returning the result. Access the bounding box coordinates via the `bbox` attribute (format `[x1, y1, x2, y2]`) and filter by width, height, or area thresholds as shown in the custom ball detector example.

### What is the difference between `query_detections` and `query_points`?

`query_detections()` (lines 61‑75 in [`dimos/models/vl/base.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/models/vl/base.py)) is optimized for bounding box outputs and uses `vlm_detection_to_detection2d()` to parse JSON arrays of `[label, x1, y1, x2, y2]`. `query_points()` uses `vlm_point_to_detection2d_point()` for single-coordinate outputs like `[label, x, y]`, returning `Detection2DPoint` objects suitable for keypoint or landmark detection tasks.

### How do agents discover and call custom detectors?

Agents discover detectors through the **`@skill`** decorator, which registers the method as a tool in the DimOS skill registry. When you define a detector method with this decorator in a module, it becomes available for LLM function calling and can be invoked via the MCP command-line interface using `dimos mcp call <skill_name>`, or automatically by agents equipped with the perception capability.