How to Add Support for New Camera Models with Custom Projections in LuisaRender

To add a custom camera in LuisaRender, inherit from the Camera base class, implement the _generate_ray_in_camera_space method in your Camera::Instance subclass, and register the plugin with LUISA_RENDER_MAKE_SCENE_NODE_PLUGIN.

LuisaRender treats cameras as scene nodes that derive from the abstract class luisa::render::Camera defined in src/base/camera.h. Adding support for new camera models with custom projections requires implementing both a host-side class for parameter parsing and a GPU-side instance class for ray generation. This guide walks through the exact implementation pattern used by built-in cameras like PinholeCamera and ThinLensCamera in the luisagroup/luisarender repository.

Understanding the Camera Architecture in LuisaRender

Every camera in LuisaRender follows a two-layer architecture. The host-side class (e.g., MyCamera) parses scene parameters from JSON, while the instance class (e.g., MyCameraInstance) executes on the GPU.

The critical virtual method you must implement is _generate_ray_in_camera_space, declared in src/base/camera.h:

[[nodiscard]] virtual std::pair<Var<Ray>, Float>
_generate_ray_in_camera_space(Expr<float2> pixel,
                              Expr<float2> u_lens,
                              Expr<float>  time) const noexcept = 0;

This function receives a pixel coordinate in image space and returns a ray in camera space along with a sampling weight. All projection math—whether perspective, orthographic, or custom fisheye—happens inside this method.

Step-by-Step Implementation Guide

Create the Camera Class and Data Structure

Create a new source file in src/cameras/ (e.g., my_camera.cpp). Begin by defining your camera class inheriting from Camera, and declare a GPU-side data struct to hold parameters:

#include <base/camera.h>
#include <base/pipeline.h>
#include <dsl/rtx/ray.h>

namespace luisa::render {

using namespace luisa::compute;

class MyCamera final : public Camera {
private:
    float _my_param;
    float2 _some_vec;

public:
    MyCamera(Scene *scene, const SceneNodeDesc *desc) noexcept
        : Camera{scene, desc},
          _my_param{desc->property_float_or_default("my_param", 50.f)},
          _some_vec{desc->property_float2_or_default("some_vec", make_float2(0.f))} {}

    [[nodiscard]] luisa::unique_ptr<Camera::Instance>
    build(Pipeline &pipeline, CommandBuffer &command_buffer) const noexcept override;

    [[nodiscard]] bool requires_lens_sampling() const noexcept override { return false; }
};

// GPU-side data layout
struct MyCameraData {
    float2 resolution;
    float  my_param;
    float2 some_vec;
};

} // namespace luisa::render

LUISA_STRUCT(luisa::render::MyCameraData, resolution, my_param, some_vec){};

The constructor reads properties from the scene description using methods like property_float_or_default. The LUISA_STRUCT macro registers the layout for GPU access.

Implement the Instance and Ray Generation

Next, implement the Camera::Instance subclass that lives on the device. In the constructor, copy host parameters to a GPU buffer. Then override _generate_ray_in_camera_space with your projection math:

namespace luisa::render {

class MyCameraInstance final : public Camera::Instance {
private:
    BufferView<MyCameraData> _device_data;

public:
    explicit MyCameraInstance(Pipeline &ppl, CommandBuffer &cb, const MyCamera *cam) noexcept
        : Camera::Instance{ppl, cb, cam},
          _device_data{ppl.arena_buffer<MyCameraData>(1u)} {
        MyCameraData host{
            make_float2(cam->film()->resolution()),
            cam->_my_param,
            cam->_some_vec};
        cb << _device_data.copy_from(&host) << commit();
    }

    [[nodiscard]] std::pair<Var<Ray>, Float>
    _generate_ray_in_camera_space(Expr<float2> pixel,
                                  Expr<float2> /*u_lens*/,
                                  Expr<float> /*time*/) const noexcept override {
        auto data = _device_data->read(0u);
        auto ndc = (pixel * 2.0f - data.resolution) / data.resolution.y;
        auto dir = normalize(make_float3(ndc.x, -ndc.y, -data.my_param * 1e-3f));
        auto ray = make_ray(make_float3(), dir);
        return {std::move(ray), 1.f};
    }
};

// Factory method
luisa::unique_ptr<Camera::Instance> MyCamera::build(
    Pipeline &pipeline, CommandBuffer &command_buffer) const noexcept {
    return luisa::make_unique<MyCameraInstance>(pipeline, command_buffer, this);
}

} // namespace luisa::render

The _generate_ray_in_camera_space method is where you implement custom projections. The example above shows a simple perspective projection, but you can replace the math with fisheye, panoramic, or arbitrary ray mappings.

Register the Plugin and Update the Build System

Wrap your camera with ClipPlaneCameraWrapper to enable optional clipping planes, then register the plugin:

namespace luisa::render {
    using ClipPlaneMyCamera = ClipPlaneCameraWrapper<MyCamera, MyCameraInstance>;
}

LUISA_RENDER_MAKE_SCENE_NODE_PLUGIN(luisa::render::ClipPlaneMyCamera)

Add the source file to src/cameras/CMakeLists.txt:

luisa_render_add_plugin(mycamera CATEGORY camera SOURCES my_camera.cpp)

This CMake function (defined in the LuisaRender build system) compiles your camera as a loadable plugin.

Using the Custom Camera in Scene Files

Reference your camera in JSON scene files using the plugin name (derived from the filename or registration):

{
    "cam": {
        "type": "camera",
        "impl": "mycamera",
        "property": {
            "resolution": [1920, 1080],
            "my_param": 35.0,
            "some_vec": [0.0, 0.0],
            "clip": [0.1, 1000.0]
        }
    },
    "render": {
        "type": "render",
        "impl": "path_tracer",
        "property": {
            "camera": "@cam"
        }
    }
}

The "clip" property works automatically because you used ClipPlaneCameraWrapper. The "impl" value must match the name passed to luisa_render_add_plugin in CMake.

Summary

• Inherit from Camera in src/cameras/my_camera.cpp to create the host-side node.
• Define a GPU data struct (e.g., MyCameraData) and register it with LUISA_STRUCT for device access.
• Implement _generate_ray_in_camera_space in your Camera::Instance subclass to define the custom projection math.
• Copy parameters to GPU buffers in the instance constructor using command_buffer << _device_data.copy_from(&host).
• Wrap with ClipPlaneCameraWrapper to support optional near/far clipping planes.
• Register with LUISA_RENDER_MAKE_SCENE_NODE_PLUGIN and add to src/cameras/CMakeLists.txt to enable JSON scene loading.

Frequently Asked Questions

What is the role of _generate_ray_in_camera_space?

The _generate_ray_in_camera_space method is the pure virtual function in src/base/camera.h that every camera instance must implement. It transforms 2D pixel coordinates into a 3D ray origin and direction in camera space, along with a sampling weight. This is where all custom projection logic—whether perspective, orthographic, or exotic lens models—must be implemented according to the LuisaRender source code.

Do I need to use the ClipPlaneCameraWrapper?

No, the ClipPlaneCameraWrapper is optional, but highly recommended. As seen in src/cameras/thin_lens.cpp and other built-in cameras, this wrapper adds support for the "clip" property in JSON scene files, allowing users to specify near and far clipping planes. Without it, your camera will still function, but will lack this standard feature available to other camera types.

How do I pass custom parameters from JSON to the GPU?

First, read the parameters in your Camera subclass constructor using methods like property_float_or_default or property_float2_or_default from the SceneNodeDesc object. Store them as member variables. Then, in your Camera::Instance constructor, pack these values into a struct (like MyCameraData), create a device buffer using pipeline.arena_buffer<>(), and copy the data using command_buffer.copy_from(). The instance can then read this data in _generate_ray_in_camera_space via BufferView::read().

Where can I find reference implementations?

Reference implementations are located in the src/cameras/ directory. src/cameras/pinhole.cpp demonstrates a basic perspective projection without lens sampling. src/cameras/thin_lens.cpp shows how to handle lens sampling and multiple GPU parameters. src/cameras/ortho.cpp provides an example of parallel projection. These files follow the exact same pattern described in this guide and serve as working templates for custom cameras.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}