# How to Use Newton's Sensor APIs for IMU, Contact Forces, and Raycasts

> Learn to use Newton's sensor APIs for IMU, contact forces, and raycasts. Access high-performance measurements and depth images with Warp-accelerated kernels.

- Repository: [Newton Physics/newton](https://github.com/newton-physics/newton)
- Tags: how-to-guide
- Published: 2026-03-19

---

**Newton's sensor APIs provide high-performance access to IMU measurements, contact forces, and depth images through Warp-accelerated kernels that automatically handle state attributes and label-based entity selection.**

Newton's sensor APIs for IMU, contact forces, and raycasts enable robotics researchers and simulation engineers to extract physically accurate data without writing custom CUDA kernels. The `newton-physics/newton` repository provides three specialized sensor classes—`SensorIMU`, `SensorContact`, and `SensorRaycast`—that integrate directly with the physics engine's state management system. These sensors leverage NVIDIA Warp for GPU acceleration while exposing simple Python APIs that handle label matching, memory allocation, and coordinate frame transformations automatically.

## Understanding Newton's Sensor Architecture

All three sensor classes follow a consistent design pattern that abstracts away the complexity of GPU kernel programming while maintaining simulation fidelity.

### Label-Based Entity Selection

Rather than requiring manual index management, sensors accept string patterns or explicit labels that resolve to body, shape, or site indices. Internally, each sensor calls `newton.utils.selection.match_labels` to map patterns like `"imu_*"` or `"ball"` to concrete simulation entities.

### Automatic State Attribute Requests

Sensors declare their data dependencies during construction to ensure memory allocation occurs before simulation begins. For example, `SensorIMU` automatically requests the `body_qdd` attribute (body accelerations), while `SensorContact` requests the `force` contact attribute. This guarantees that `model.state()` and `model.contacts()` allocate the required Warp arrays.

### Warp-Accelerated Computation

The computational heavy lifting occurs in small, specialized Warp kernels that execute on CPU or GPU. The Python wrapper classes handle input preparation, kernel launches, and result conversion to `wp.array` or NumPy views, eliminating the need for manual CUDA programming.

## Measuring Specific Force with SensorIMU

The `SensorIMU` class in [`newton/_src/sensors/sensor_imu.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/sensors/sensor_imu.py) measures linear accelerations and angular velocities at arbitrary sites within the simulation, accounting for gravity, Coriolis effects, and body accelerations.

### IMU Kernel Implementation

The `compute_sensor_imu_kernel` function (lines 26-77) transforms site positions to body frames, computes specific force by combining linear acceleration, gravity subtraction, and Coriolis terms (`cross(ω, cross(ω, r))`), then rotates results into the sensor's local coordinate frame.

### Code Example: Reading Accelerometer and Gyroscope Data

```python
import warp as wp
import newton
from newton.sensors import SensorIMU

# Build a model with an IMU site

builder = newton.ModelBuilder()
builder.add_ground_plane()
body = builder.add_body(xform=wp.transform((0, 0, 0.5), wp.quat_identity()))
builder.add_shape_sphere(body, radius=0.1, label="ball")
builder.add_site(body, label="imu_0")

model = builder.finalize()

# Create sensor using label pattern matching

imu = SensorIMU(model, sites="imu_*")

# Simulate and update

solver = newton.solvers.SolverMuJoCo(model)
state = model.state()
solver.step(state, state, None, None, dt=1/240)

imu.update(state)

# Retrieve measurements as NumPy arrays

accelerometer = imu.accelerometer.numpy()  # Shape: (n_sensors, 3) [m/s²]

gyroscope = imu.gyroscope.numpy()        # Shape: (n_sensors, 3) [rad/s]

print(f"Accelerometer: {accelerometer}")
print(f"Gyroscope: {gyroscope}")

```

*The sensor automatically requests the `body_qdd` attribute during construction (lines 60-63 in [`sensor_imu.py`](https://github.com/newton-physics/newton/blob/main/sensor_imu.py)), ensuring the state object contains acceleration data.*

## Monitoring Contact Forces with SensorContact

The `SensorContact` class in [`newton/_src/sensors/sensor_contact.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/sensors/sensor_contact.py) aggregates net contact forces on bodies or shapes, optionally breaking down forces by counterpart entities.

### How Contact Aggregation Works

The implementation uses an `ObjectType` enum (lines 41-52) to distinguish between **TOTAL**, **SHAPE**, and **BODY** aggregation modes. The `_assemble_sensor_mappings` static method (lines 58-86) builds compact shape-pair to reading-index mappings, pruning non-colliding pairs using the model's `shape_contact_pairs` set to minimize computation.

The `select_aggregate_net_force_kernel` (lines 75-146) reads the contact force array (`contacts.force`) and atomically accumulates forces for each sensor-counterpart pair.

### Code Example: Monitoring Shape-to-Shape Contact Forces

```python
import warp as wp
import newton
from newton.sensors import SensorContact

# Create a ball-on-ground scenario

builder = newton.ModelBuilder()
builder.add_ground_plane()
body = builder.add_body(xform=wp.transform((0, 0, 0.1), wp.quat_identity()))
builder.add_shape_sphere(body, radius=0.1, label="ball")
model = builder.finalize()

# Initialize contact sensor with label matching

contact_sensor = SensorContact(
    model,
    sensing_obj_shapes="ball",
    counterpart_shapes="ground_plane",
    include_total=True
)

solver = newton.solvers.SolverMuJoCo(model)
state = model.state()
contacts = model.contacts()

# Simulation step with contact update

solver.step(state, state, None, None, dt=1/240)
solver.update_contacts(contacts)

# Update the sensor

contact_sensor.update(state, contacts)

# Net forces (shape: (n_sensors, max_readings, 3))

forces = contact_sensor.net_force.numpy()
print(f"Contact forces shape: {forces.shape}")
print(f"Net contact force (N): {forces[0, 0]}")

```

*`SensorContact` automatically requests the `force` contact attribute (lines 176-179 in [`sensor_contact.py`](https://github.com/newton-physics/newton/blob/main/sensor_contact.py)). If you instantiate `Contacts` before the sensor, manually request the attribute via `model.request_contact_attributes(['force'])`.*

## Generating Depth Images with SensorRaycast

The `SensorRaycast` class in [`newton/_src/sensors/sensor_raycast.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/sensors/sensor_raycast.py) generates depth images by casting rays from a virtual camera into the simulation world, supporting both geometric shapes and particle systems.

### Camera Model and Ray Casting

The sensor implements a pinhole camera model with configurable field-of-view, aspect ratio, and resolution. The `_compute_camera_basis` method (lines 27-54) orthonormalizes the camera's forward, up, and right vectors to ensure consistent ray directions across the image plane.

The `sensor_raycast_kernel` (imported from `newton.geometry.raycast`) executes per-pixel ray-shape intersection tests, returning the nearest hit distance. For particle systems, the sensor utilizes a spatial hash grid (`wp.HashGrid`) with marching steps bounded by `MAX_PARTICLE_RAY_MARCH_STEPS` (line 35) to prevent infinite loops. The `clamp_no_hits_kernel` (lines 26-33) post-processes results, replacing `max_distance` values with `-1.0` to indicate no intersection occurred.

### Code Example: Creating a Virtual Depth Camera

```python
import warp as wp
import numpy as np
import newton
from newton.sensors import SensorRaycast

# Build a simple scene

builder = newton.ModelBuilder()
builder.add_ground_plane()
box = builder.add_body(xform=wp.transform((0, 0, 0.5), wp.quat_identity()))
builder.add_shape_box(box, half_extents=(0.2, 0.2, 0.2), label="box")
model = builder.finalize()

# Configure depth sensor looking downward

raycast = SensorRaycast(
    model,
    camera_position=(0.0, 0.0, 2.0),
    camera_direction=(0.0, 0.0, -1.0),
    camera_up=(0.0, 1.0, 0.0),
    fov_radians=np.radians(60.0),
    width=320,
    height=240,
    max_distance=5.0
)

# Run a simulation step (optional if bodies move)

solver = newton.solvers.SolverMuJoCo(model)
state = model.state()
solver.step(state, state, None, None, dt=1/120)

# Generate a depth image

raycast.update(state)
depth_image = raycast.get_depth_image_numpy()

print(f"Depth image shape: {depth_image.shape}")
print(f"Center pixel depth: {depth_image[120, 160]} meters")

```

*Update camera poses at runtime using `update_camera_pose` or `point_camera_at`. To include particles in ray casting, call `update(state, include_particles=True, particle_march_step=0.01)`.*

## Key Implementation Files

The sensor system spans multiple modules in the Newton repository:

| File | Purpose |
|------|---------|
| [`newton/_src/sensors/sensor_imu.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/sensors/sensor_imu.py) | `SensorIMU` implementation with `compute_sensor_imu_kernel` for specific force calculations |
| [`newton/_src/sensors/sensor_contact.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/sensors/sensor_contact.py) | `SensorContact` with `select_aggregate_net_force_kernel` and `ObjectType` enum |
| [`newton/_src/sensors/sensor_raycast.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/sensors/sensor_raycast.py) | `SensorRaycast` camera model and particle ray-marching logic |
| [`newton/_src/geometry/raycast.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/geometry/raycast.py) | Low-level `sensor_raycast_kernel` for geometric intersection tests |
| [`newton/_src/sim/state.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/sim/state.py) | `State` class definition containing `body_qdd` and other extended attributes |
| [`newton/_src/sim/contacts.py`](https://github.com/newton-physics/newton/blob/main/newton/_src/sim/contacts.py) | `Contacts` class with `force` attribute for contact sensors |
| [`newton/utils/selection.py`](https://github.com/newton-physics/newton/blob/main/newton/utils/selection.py) | `match_labels` utility for pattern-based entity resolution |

## Best Practices for Sensor Performance

To maximize throughput when utilizing Newton's sensor APIs:

- **Batch sensor updates** – Update all sensors sequentially after each solver step rather than interleaving updates with simulation substeps. The underlying Warp kernels are lightweight and share device contexts efficiently.
- **Match device contexts** – Ensure the model's compute device matches your execution target. Set `model.device = wp.device('gpu:0')` for GPU acceleration, or use the default CPU device for debugging.
- **Minimize particle overhead** – When using `SensorRaycast`, enable `include_particles=True` only when your simulation actually contains particle systems. The spatial hash grid construction adds unnecessary overhead for purely rigid-body scenes.
- **Debug with NumPy views** – Access the underlying Warp arrays directly (`imu.accelerometer`, `contact_sensor.net_force`, `raycast.depth_image`) and convert to NumPy using `.numpy()` for visualization with standard tools like Matplotlib or OpenCV.

## Summary

- **Newton's sensor APIs** provide ready-made solutions for extracting IMU data, contact forces, and depth images without custom CUDA kernel development.
- **SensorIMU** computes specific force and angular velocity at arbitrary sites using the `compute_sensor_imu_kernel` in [`sensor_imu.py`](https://github.com/newton-physics/newton/blob/main/sensor_imu.py), automatically handling gravity and Coriolis effects.
- **SensorContact** aggregates net contact forces via atomic operations in `select_aggregate_net_force_kernel`, supporting both total force measurements and per-counterpart breakdowns.
- **SensorRaycast** generates depth images using geometric ray casting with optional particle intersection via spatial hashing, implemented in [`sensor_raycast.py`](https://github.com/newton-physics/newton/blob/main/sensor_raycast.py).
- All sensors utilize **automatic attribute requests** to ensure `State` and `Contacts` objects allocate required buffers (`body_qdd`, `force`) before simulation begins.

## Frequently Asked Questions

### How do I add an IMU sensor to a specific body in Newton?

Attach a site to your body using `builder.add_site(body, label="imu_site")`, then instantiate `SensorIMU(model, sites="imu_site")` or use wildcard patterns like `"imu_*"` to match multiple sites. The sensor automatically requests the `body_qdd` attribute from the model to access body accelerations during updates.

### Can SensorContact measure forces between specific shape pairs?

Yes. Pass explicit labels to the `sensing_obj_shapes` and `counterpart_shapes` parameters when constructing `SensorContact`. The internal `_assemble_sensor_mappings` method filters the model's `shape_contact_pairs` set to include only relevant collisions, and the `select_aggregate_net_force_kernel` aggregates forces separately for each valid pair.

### What coordinate frame does SensorRaycast use for depth values?

`SensorRaycast` returns distances in world coordinates (meters) along each camera ray. The `_compute_camera_basis` method orthonormalizes the camera's forward, up, and right vectors to ensure consistent ray directions. Depth values equal to `max_distance` are clamped to `-1.0` by `clamp_no_hits_kernel` to indicate no intersection occurred.

### Do I need to manually allocate memory for sensor data?

No. Newton's sensors automatically request required extended attributes during construction. `SensorIMU` requests `body_qdd` for accelerations, `SensorContact` requests the `force` contact attribute, and `SensorRaycast` allocates internal buffers for depth images. Simply call `model.state()` and `model.contacts()` after sensor instantiation to allocate the necessary Warp arrays.