How Collision Filtering Using Category and Mask Bits Works in Box3D

Box3D determines whether two shapes collide by testing reciprocal 64-bit category and mask bitmasks, with an optional group index that can override bitwise logic to force or disable collisions for specific object groups.

Collision filtering using category and mask bits in the erincatto/box3d repository provides precise control over which physics objects generate contacts. The system evaluates potential collisions early using efficient bitwise operations defined in src/shape.h, preventing expensive narrow-phase detection between objects that should not interact, such as teammates or projectiles and their owners.

Understanding the b3Filter Structure

The collision filter in Box3D centers on the b3Filter structure defined in include/box3d/types.h. This structure contains three fields that govern interaction eligibility:

• categoryBits: A 64-bit bitmask declaring which categories the shape belongs to.
• maskBits: A 64-bit bitmask declaring which categories this shape will accept collisions from.
• groupIndex: A signed integer that overrides category/mask logic when two shapes share the same non-zero value.

When you initialize a shape definition with b3DefaultShapeDef(), the filter defaults to categoryBits = 1 (category 1) and maskBits = B3_DEFAULT_MASK_BITS (collides with everything). You override these values by setting b3ShapeDef.filter before creating the shape.

The Collision Decision Logic in src/shape.h

The core collision test resides in the inline function b3ShouldShapesCollide within src/shape.h. This function returns a boolean indicating whether two shapes should proceed to narrow-phase contact generation.

Group Index Overrides

The function first checks for group-based exceptions before evaluating category bits:

if (filterA.groupIndex == filterB.groupIndex && filterA.groupIndex != 0)
{
    return filterA.groupIndex > 0;
}

When two shapes share an identical non-zero groupIndex, the sign of that integer dictates the result immediately. A positive value forces collision between the shapes regardless of their category settings, while a negative value unconditionally disables it. This mechanism is ideal for implementing team-based collision rules or ensuring that projectiles always hit their targets.

Category and Mask Bitwise Test

If no group override applies, Box3D performs a reciprocal bitwise AND test:

return (filterA.maskBits & filterB.categoryBits) != 0 &&
       (filterA.categoryBits & filterB.maskBits) != 0;

For a collision to occur, both conditions must evaluate to true. Shape A must have a category bit that Shape B's mask accepts, and Shape B must have a category bit that Shape A's mask accepts. This bidirectional requirement ensures mutual collision consent.

The engine applies identical logic to shape-versus-query tests (such as ray casts) through b3ShouldQueryCollide, allowing spatial queries to respect the same filtering rules as physical collisions.

Configuring Collision Filters in Practice

You configure collision behavior by populating the filter field on b3ShapeDef before calling creation functions like b3CreateBoxShape or b3CreateSphereShape.

Category-to-Category Filtering

To establish specific collision relationships between object types, set the category and mask bits to target specific bit positions:

b3ShapeDef defA = b3DefaultShapeDef();
defA.filter.categoryBits = 0x01;   // Belongs to category 1
defA.filter.maskBits     = 0x02;   // Accepts collisions from category 2
b3CreateBoxShape(world, bodyA, &defA, &boxA);

b3ShapeDef defB = b3DefaultShapeDef();
defB.filter.categoryBits = 0x02;   // Belongs to category 2
defB.filter.maskBits     = 0x01;   // Accepts collisions from category 1
b3CreateBoxShape(world, bodyB, &defB, &boxB);

These shapes will collide because (0x02 & 0x02) != 0 and (0x01 & 0x01) != 0, satisfying both reciprocal bitwise conditions.

Group-Based Team Collision

Use groupIndex to bypass category logic for entire teams or projectile groups:

b3ShapeDef playerDef = b3DefaultShapeDef();
playerDef.filter.groupIndex = -1;  // Same negative group: never collide
b3CreateBoxShape(world, playerBody1, &playerDef, &player1);
b3CreateBoxShape(world, playerBody2, &playerDef, &player2);

b3ShapeDef bulletDef = b3DefaultShapeDef();
bulletDef.filter.groupIndex = 1;   // Positive group: always collide
b3CreateSphereShape(world, bulletBody, &bulletDef, &bullet);

In this configuration, the two players pass through each other despite their category settings, while bullets sharing the positive group index will always collide with one another.

Query Filtering for Ray Casts

Ray casts and overlap queries respect the same filtering mechanism through b3QueryFilter:

b3QueryFilter query = {
    .categoryBits = 0x01,  // Query belongs to category 1
    .maskBits = 0xFF       // Accepts collisions with categories 0-7
};
b3RayCast(world, start, direction, maxDistance, &query, callback);

This setup ensures the ray only reports hits against shapes whose categories are enabled in the query's mask, and whose masks accept category 1.

Performance Benefits of Early Rejection

Collision filtering using category and mask bits occurs during the broad phase in src/broad_phase.c before expensive narrow-phase calculations begin. By inserting objects into the dynamic tree with their categoryBits, the engine culls impossible collision pairs early using the trivial bitwise operations in b3ShouldShapesCollide. This prevents the generation of contact manifolds and GJK/EPA calculations for objects that are logically unrelated, which is essential for maintaining high performance in simulations with thousands of non-interacting entities.

Summary

• Box3D collision filtering relies on 64-bit category and mask bitmasks defined in b3Filter and evaluated in src/shape.h.
• The b3ShouldShapesCollide function requires both (maskA & categoryB) and (categoryA & maskB) to be non-zero for collision approval.
• Group index overrides category/mask logic when two shapes share the same non-zero value: positive forces collision, negative disables it.
• Configure filters via b3ShapeDef.filter before calling b3CreateBoxShape, b3CreateSphereShape, or similar creation functions.
• Query operations use identical filtering through b3QueryFilter, ensuring ray casts respect shape collision rules.
• Early filtering in the broad phase prevents unnecessary narrow-phase computations, optimizing simulation performance.

Frequently Asked Questions

What happens if two shapes have no overlapping category and mask bits?

If either (maskA & categoryB) or (categoryA & maskB) equals zero, b3ShouldShapesCollide returns false and the potential collision is discarded during the broad phase. The objects will pass through each other without generating contacts or invoking contact listeners.

Can a single shape belong to multiple collision categories?

Yes. Since categoryBits is a 64-bit field, you can assign a shape to multiple categories by OR-ing bit values: shapeDef.filter.categoryBits = 0x01 | 0x04 | 0x08. The shape will collide with any query or shape whose maskBits has any of those corresponding bits set.

How does the group index interact with category bits?

When two shapes share an identical non-zero groupIndex, the group logic takes precedence and the category/mask bits are ignored entirely. A positive group index forces collision; a negative group index prevents it. This is useful for creating teams or collision exceptions without reconfiguring individual category masks for every shape.

Where is the collision filter data stored in the Box3D source?

The filter structure b3Filter is defined in include/box3d/types.h. The collision decision logic resides in src/shape.h within the b3ShouldShapesCollide and b3ShouldQueryCollide functions. Shape creation and default filter application occur in src/shape.c, while the broad phase uses category bits when inserting proxies into the dynamic tree in src/broad_phase.c.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}