How to Use Newton's Sensor APIs for IMU, Contact Forces, and Raycasts

Newton's sensor APIs provide high-performance access to IMU measurements, contact forces, and depth images through Warp-accelerated kernels that automatically handle state attributes and label-based entity selection.

Newton's sensor APIs for IMU, contact forces, and raycasts enable robotics researchers and simulation engineers to extract physically accurate data without writing custom CUDA kernels. The newton-physics/newton repository provides three specialized sensor classes—SensorIMU, SensorContact, and SensorRaycast—that integrate directly with the physics engine's state management system. These sensors leverage NVIDIA Warp for GPU acceleration while exposing simple Python APIs that handle label matching, memory allocation, and coordinate frame transformations automatically.

Understanding Newton's Sensor Architecture

All three sensor classes follow a consistent design pattern that abstracts away the complexity of GPU kernel programming while maintaining simulation fidelity.

Label-Based Entity Selection

Rather than requiring manual index management, sensors accept string patterns or explicit labels that resolve to body, shape, or site indices. Internally, each sensor calls newton.utils.selection.match_labels to map patterns like "imu_*" or "ball" to concrete simulation entities.

Automatic State Attribute Requests

Sensors declare their data dependencies during construction to ensure memory allocation occurs before simulation begins. For example, SensorIMU automatically requests the body_qdd attribute (body accelerations), while SensorContact requests the force contact attribute. This guarantees that model.state() and model.contacts() allocate the required Warp arrays.

Warp-Accelerated Computation

The computational heavy lifting occurs in small, specialized Warp kernels that execute on CPU or GPU. The Python wrapper classes handle input preparation, kernel launches, and result conversion to wp.array or NumPy views, eliminating the need for manual CUDA programming.

Measuring Specific Force with SensorIMU

The SensorIMU class in newton/_src/sensors/sensor_imu.py measures linear accelerations and angular velocities at arbitrary sites within the simulation, accounting for gravity, Coriolis effects, and body accelerations.

IMU Kernel Implementation

The compute_sensor_imu_kernel function (lines 26-77) transforms site positions to body frames, computes specific force by combining linear acceleration, gravity subtraction, and Coriolis terms (cross(ω, cross(ω, r))), then rotates results into the sensor's local coordinate frame.

Code Example: Reading Accelerometer and Gyroscope Data

import warp as wp
import newton
from newton.sensors import SensorIMU

# Build a model with an IMU site

builder = newton.ModelBuilder()
builder.add_ground_plane()
body = builder.add_body(xform=wp.transform((0, 0, 0.5), wp.quat_identity()))
builder.add_shape_sphere(body, radius=0.1, label="ball")
builder.add_site(body, label="imu_0")

model = builder.finalize()

# Create sensor using label pattern matching

imu = SensorIMU(model, sites="imu_*")

# Simulate and update

solver = newton.solvers.SolverMuJoCo(model)
state = model.state()
solver.step(state, state, None, None, dt=1/240)

imu.update(state)

# Retrieve measurements as NumPy arrays

accelerometer = imu.accelerometer.numpy()  # Shape: (n_sensors, 3) [m/s²]

gyroscope = imu.gyroscope.numpy()        # Shape: (n_sensors, 3) [rad/s]

print(f"Accelerometer: {accelerometer}")
print(f"Gyroscope: {gyroscope}")

The sensor automatically requests the body_qdd attribute during construction (lines 60-63 in sensor_imu.py), ensuring the state object contains acceleration data.

Monitoring Contact Forces with SensorContact

The SensorContact class in newton/_src/sensors/sensor_contact.py aggregates net contact forces on bodies or shapes, optionally breaking down forces by counterpart entities.

How Contact Aggregation Works

The implementation uses an ObjectType enum (lines 41-52) to distinguish between TOTAL, SHAPE, and BODY aggregation modes. The _assemble_sensor_mappings static method (lines 58-86) builds compact shape-pair to reading-index mappings, pruning non-colliding pairs using the model's shape_contact_pairs set to minimize computation.

The select_aggregate_net_force_kernel (lines 75-146) reads the contact force array (contacts.force) and atomically accumulates forces for each sensor-counterpart pair.

Code Example: Monitoring Shape-to-Shape Contact Forces

import warp as wp
import newton
from newton.sensors import SensorContact

# Create a ball-on-ground scenario

builder = newton.ModelBuilder()
builder.add_ground_plane()
body = builder.add_body(xform=wp.transform((0, 0, 0.1), wp.quat_identity()))
builder.add_shape_sphere(body, radius=0.1, label="ball")
model = builder.finalize()

# Initialize contact sensor with label matching

contact_sensor = SensorContact(
    model,
    sensing_obj_shapes="ball",
    counterpart_shapes="ground_plane",
    include_total=True
)

solver = newton.solvers.SolverMuJoCo(model)
state = model.state()
contacts = model.contacts()

# Simulation step with contact update

solver.step(state, state, None, None, dt=1/240)
solver.update_contacts(contacts)

# Update the sensor

contact_sensor.update(state, contacts)

# Net forces (shape: (n_sensors, max_readings, 3))

forces = contact_sensor.net_force.numpy()
print(f"Contact forces shape: {forces.shape}")
print(f"Net contact force (N): {forces[0, 0]}")

SensorContact automatically requests the force contact attribute (lines 176-179 in sensor_contact.py). If you instantiate Contacts before the sensor, manually request the attribute via model.request_contact_attributes(['force']).

Generating Depth Images with SensorRaycast

The SensorRaycast class in newton/_src/sensors/sensor_raycast.py generates depth images by casting rays from a virtual camera into the simulation world, supporting both geometric shapes and particle systems.

Camera Model and Ray Casting

The sensor implements a pinhole camera model with configurable field-of-view, aspect ratio, and resolution. The _compute_camera_basis method (lines 27-54) orthonormalizes the camera's forward, up, and right vectors to ensure consistent ray directions across the image plane.

The sensor_raycast_kernel (imported from newton.geometry.raycast) executes per-pixel ray-shape intersection tests, returning the nearest hit distance. For particle systems, the sensor utilizes a spatial hash grid (wp.HashGrid) with marching steps bounded by MAX_PARTICLE_RAY_MARCH_STEPS (line 35) to prevent infinite loops. The clamp_no_hits_kernel (lines 26-33) post-processes results, replacing max_distance values with -1.0 to indicate no intersection occurred.

Code Example: Creating a Virtual Depth Camera

import warp as wp
import numpy as np
import newton
from newton.sensors import SensorRaycast

# Build a simple scene

builder = newton.ModelBuilder()
builder.add_ground_plane()
box = builder.add_body(xform=wp.transform((0, 0, 0.5), wp.quat_identity()))
builder.add_shape_box(box, half_extents=(0.2, 0.2, 0.2), label="box")
model = builder.finalize()

# Configure depth sensor looking downward

raycast = SensorRaycast(
    model,
    camera_position=(0.0, 0.0, 2.0),
    camera_direction=(0.0, 0.0, -1.0),
    camera_up=(0.0, 1.0, 0.0),
    fov_radians=np.radians(60.0),
    width=320,
    height=240,
    max_distance=5.0
)

# Run a simulation step (optional if bodies move)

solver = newton.solvers.SolverMuJoCo(model)
state = model.state()
solver.step(state, state, None, None, dt=1/120)

# Generate a depth image

raycast.update(state)
depth_image = raycast.get_depth_image_numpy()

print(f"Depth image shape: {depth_image.shape}")
print(f"Center pixel depth: {depth_image[120, 160]} meters")

Update camera poses at runtime using update_camera_pose or point_camera_at. To include particles in ray casting, call update(state, include_particles=True, particle_march_step=0.01).

Key Implementation Files

The sensor system spans multiple modules in the Newton repository:

File	Purpose
newton/_src/sensors/sensor_imu.py	SensorIMU implementation with compute_sensor_imu_kernel for specific force calculations
newton/_src/sensors/sensor_contact.py	SensorContact with select_aggregate_net_force_kernel and ObjectType enum
newton/_src/sensors/sensor_raycast.py	SensorRaycast camera model and particle ray-marching logic
newton/_src/geometry/raycast.py	Low-level sensor_raycast_kernel for geometric intersection tests
newton/_src/sim/state.py	State class definition containing body_qdd and other extended attributes
newton/_src/sim/contacts.py	Contacts class with force attribute for contact sensors
newton/utils/selection.py	match_labels utility for pattern-based entity resolution

Best Practices for Sensor Performance

To maximize throughput when utilizing Newton's sensor APIs:

• Batch sensor updates – Update all sensors sequentially after each solver step rather than interleaving updates with simulation substeps. The underlying Warp kernels are lightweight and share device contexts efficiently.
• Match device contexts – Ensure the model's compute device matches your execution target. Set model.device = wp.device('gpu:0') for GPU acceleration, or use the default CPU device for debugging.
• Minimize particle overhead – When using SensorRaycast, enable include_particles=True only when your simulation actually contains particle systems. The spatial hash grid construction adds unnecessary overhead for purely rigid-body scenes.
• Debug with NumPy views – Access the underlying Warp arrays directly (imu.accelerometer, contact_sensor.net_force, raycast.depth_image) and convert to NumPy using .numpy() for visualization with standard tools like Matplotlib or OpenCV.

Summary

• Newton's sensor APIs provide ready-made solutions for extracting IMU data, contact forces, and depth images without custom CUDA kernel development.
• SensorIMU computes specific force and angular velocity at arbitrary sites using the compute_sensor_imu_kernel in sensor_imu.py, automatically handling gravity and Coriolis effects.
• SensorContact aggregates net contact forces via atomic operations in select_aggregate_net_force_kernel, supporting both total force measurements and per-counterpart breakdowns.
• SensorRaycast generates depth images using geometric ray casting with optional particle intersection via spatial hashing, implemented in sensor_raycast.py.
• All sensors utilize automatic attribute requests to ensure State and Contacts objects allocate required buffers (body_qdd, force) before simulation begins.

Frequently Asked Questions

How do I add an IMU sensor to a specific body in Newton?

Attach a site to your body using builder.add_site(body, label="imu_site"), then instantiate SensorIMU(model, sites="imu_site") or use wildcard patterns like "imu_*" to match multiple sites. The sensor automatically requests the body_qdd attribute from the model to access body accelerations during updates.

Can SensorContact measure forces between specific shape pairs?

Yes. Pass explicit labels to the sensing_obj_shapes and counterpart_shapes parameters when constructing SensorContact. The internal _assemble_sensor_mappings method filters the model's shape_contact_pairs set to include only relevant collisions, and the select_aggregate_net_force_kernel aggregates forces separately for each valid pair.

What coordinate frame does SensorRaycast use for depth values?

SensorRaycast returns distances in world coordinates (meters) along each camera ray. The _compute_camera_basis method orthonormalizes the camera's forward, up, and right vectors to ensure consistent ray directions. Depth values equal to max_distance are clamped to -1.0 by clamp_no_hits_kernel to indicate no intersection occurred.

Do I need to manually allocate memory for sensor data?

No. Newton's sensors automatically request required extended attributes during construction. SensorIMU requests body_qdd for accelerations, SensorContact requests the force contact attribute, and SensorRaycast allocates internal buffers for depth images. Simply call model.state() and model.contacts() after sensor instantiation to allocate the necessary Warp arrays.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}