How to Customize or Extend the GLRecordingPipeline for Custom Effects in FadCam

You can customize the GLRecordingPipeline in FadCam by extending the GLWatermarkRenderer class to inject custom OpenGL shaders into the render loop while the pipeline handles encoding and frame synchronization automatically.

The FadCam open-source camera app uses a hardware-accelerated recording pipeline to capture video on Android. If you want to customize or extend the GLRecordingPipeline for custom effects like grayscale filters, LUTs, or real-time overlays, you will work primarily with the GLWatermarkRenderer class, leaving the core GLRecordingPipeline orchestration unchanged.

Architecture Overview

FadCam’s recording stack separates concerns between pipeline orchestration and rendering implementation. Understanding this separation is critical for injecting custom visual effects without breaking audio-video synchronization.

Core Components

The system relies on two primary classes in the app/src/main/java/com/fadcam/opengl/ package:

• GLRecordingPipeline (GLRecordingPipeline.java): Manages the MediaCodec encoder, EGL context lifecycle, audio synchronization, and segmented file output. It instantiates the renderer via prepareSurfaces() and provides the SegmentCallback interface for file-roll events.
• GLWatermarkRenderer (GLWatermarkRenderer.java): Owns the OpenGL shader programs, the OES external texture that receives camera frames, and the renderToEncoderInternal() method that executes the per-frame draw path.

Data Flow

Camera frames flow through the system in this order:

1. Camera2 API writes frames to the OES texture (oesTextureId) managed by GLWatermarkRenderer.
2. renderToEncoderInternal() executes on the dedicated render thread, updating texture matrices and applying exposure compensation.
3. Custom effects can be drawn after the texture update but before the watermark and PiP overlays.
4. eglSwapBuffers sends the final frame to the encoderInputSurface, which feeds the MediaCodec encoder inside GLRecordingPipeline.

Because the pipeline guarantees an active EGL context during renderToEncoderInternal(), any OpenGL calls you add will execute safely within the encoder’s thread.

Extension Points for Custom Effects

The renderer exposes three specific hooks for customization: shader creation, the render loop insertion point, and the segment callback for dynamic effects.

Shader Creation in GLWatermarkRenderer

Custom effects require compiled OpenGL ES 2.0 shader programs. The renderer already initializes shaders in initializeEGL(), making it the logical place to load your custom programs.

You will add your fragment shader source as a constant, then compile and link it using the existing loadShader() helper method.

The Render Loop Hook

The renderToEncoderInternal(boolean allowStaleFrame) method around line 540 of GLWatermarkRenderer.java is the primary injection point. After the texture matrix updates (which handle exposure and orientation), you can insert draw calls that process the oesTextureId through your custom shader before the existing watermark and PiP draw calls.

Runtime Control via SegmentCallback

For effects that change per file segment (such as alternating LUTs or color grades), implement GLRecordingPipeline.SegmentCallback when constructing the pipeline in RecordingService.java. The callback receives the next segment number, allowing you to update renderer uniforms or swap textures atomically at file boundaries.

Implementation Guide: Adding a Grayscale Filter

Follow these steps to implement a runtime-togglable grayscale effect that processes frames before the watermark overlay.

Step 1: Define the Fragment Shader

Add the grayscale fragment shader to GLWatermarkRenderer.java near the existing shader constants:

private static final String GRAYSCALE_FRAGMENT_SHADER =
        "precision mediump float;\n" +
        "varying vec2 vTexCoord;\n" +
        "uniform samplerExternalOES sTexture;\n" +
        "void main() {\n" +
        "    vec4 rgba = texture2D(sTexture, vTexCoord);\n" +
        "    float gray = dot(rgba.rgb, vec3(0.299, 0.587, 0.114));\n" +
        "    gl_FragColor = vec4(gray, gray, gray, rgba.a);\n" +
        "}";

Step 2: Compile the Shader Program

Declare handles for your custom program and initialize them in initializeEGL() after setupOESShader():

private int customEffectProgram = 0;
private int customEffectPositionHandle;
private int customEffectTexCoordHandle;
private int customEffectTextureHandle;
private boolean grayscaleEnabled = false;

private void setupCustomEffectShader() {
    int vertex = loadShader(GLES20.GL_VERTEX_SHADER, VERTEX_SHADER); // Reuse existing vertex shader
    int fragment = loadShader(GLES20.GL_FRAGMENT_SHADER, GRAYSCALE_FRAGMENT_SHADER);
    customEffectProgram = GLES20.glCreateProgram();
    GLES20.glAttachShader(customEffectProgram, vertex);
    GLES20.glAttachShader(customEffectProgram, fragment);
    GLES20.glLinkProgram(customEffectProgram);
    
    customEffectPositionHandle = GLES20.glGetAttribLocation(customEffectProgram, "aPosition");
    customEffectTexCoordHandle = GLES20.glGetAttribLocation(customEffectProgram, "aTexCoord");
    customEffectTextureHandle = GLES20.glGetUniformLocation(customEffectProgram, "sTexture");
}

Step 3: Inject into the Render Loop

Modify renderToEncoderInternal() to execute your custom draw calls after updateMatrices() but before the watermark rendering:

// Inside renderToEncoderInternal(), after updateMatrices():
if (grayscaleEnabled && customEffectProgram != 0) {
    GLES20.glUseProgram(customEffectProgram);
    GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
    GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, oesTextureId);
    GLES20.glUniform1i(customEffectTextureHandle, 0);

    vertexBuffer.position(0);
    GLES20.glVertexAttribPointer(customEffectPositionHandle, 2,
            GLES20.GL_FLOAT, false, 0, vertexBuffer);
    GLES20.glEnableVertexAttribArray(customEffectPositionHandle);

    texCoordBuffer.position(0);
    GLES20.glVertexAttribPointer(customEffectTexCoordHandle, 2,
            GLES20.GL_FLOAT, false, 0, texCoordBuffer);
    GLES20.glEnableVertexAttribArray(customEffectTexCoordHandle);

    GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4);

    GLES20.glDisableVertexAttribArray(customEffectPositionHandle);
    GLES20.glDisableVertexAttribArray(customEffectTexCoordHandle);
}

Step 4: Expose Runtime Control

Add a public method to toggle the effect and expose a getter in GLRecordingPipeline if needed:

public void setGrayscaleEnabled(boolean enabled) {
    if (enabled && customEffectProgram == 0) {
        setupCustomEffectShader();
    }
    this.grayscaleEnabled = enabled;
}

In RecordingService.java, access the renderer through the pipeline reference:

GLWatermarkRenderer renderer = pipeline.getRenderer(); // Add getRenderer() to GLRecordingPipeline if missing
renderer.setGrayscaleEnabled(true);

Step 5: Implement Per-Segment Effects

To change effects when the video rolls over to a new segment, pass a SegmentCallback when constructing the pipeline:

GLRecordingPipeline.SegmentCallback callback = nextSegment -> {
    if (renderer != null) {
        renderer.loadLutForSegment(nextSegment); // Custom method you implement
    }
};

GLRecordingPipeline pipeline = new GLRecordingPipeline(
        context,
        watermarkInfoProvider,
        width,
        height,
        framerate,
        outputPath,
        maxFileSize,
        1,
        callback,  // Segment callback
        previewSurface,
        orientation,
        sensorOrientation,
        videoCodec,
        latitude,
        longitude);

Summary

• Extend via GLWatermarkRenderer: Keep GLRecordingPipeline unchanged; add shaders and draw logic in GLWatermarkRenderer.java.
• Target renderToEncoderInternal(): Insert custom GL calls after matrix updates but before watermark/PiP rendering to ensure effects apply to the base frame.
• Maintain GL Context Safety: All custom OpenGL calls execute within the pipeline’s guaranteed EGL context, so you can safely bind textures and shaders without additional synchronization.
• Use SegmentCallback: Implement this interface to trigger effect changes at file boundaries, perfect for multi-segment recordings with varying color grades.
• Access via RecordingService: Instantiate or configure your custom renderer in RecordingService.java, which constructs the pipeline and manages the camera session lifecycle.

Frequently Asked Questions

What is the GLRecordingPipeline responsible for in FadCam?

The GLRecordingPipeline class in app/src/main/java/com/fadcam/opengl/GLRecordingPipeline.java manages the entire video encoding lifecycle, including MediaCodec configuration, EGL context creation on a background thread, audio-video timestamp synchronization, and segmented MP4 muxing via FragmentedMp4MuxerWrapper. It delegates all visual rendering to GLWatermarkRenderer but handles the final transport of encoded buffers to disk.

Where should I add custom shader code in GLWatermarkRenderer?

Add custom shader source code as String constants near the existing shader definitions (around line 180), compile them in initializeEGL() after the OES shader setup, and invoke the resulting program inside renderToEncoderInternal() after the texture matrix updates (around line 540). This ensures your effect processes the camera frame before the watermark and PiP overlays are drawn.

How do I toggle effects during recording?

Expose a public setter method in GLWatermarkRenderer (such as setGrayscaleEnabled(boolean)) that updates a boolean flag checked inside renderToEncoderInternal(). Access the renderer instance through GLRecordingPipeline by adding a getRenderer() getter, then call your setter from RecordingService.java or any UI controller that holds the pipeline reference.

Can I apply different effects per video segment?

Yes. Implement GLRecordingPipeline.SegmentCallback and pass it to the pipeline constructor in RecordingService.java. The onSegmentChanged(int nextSegment) method fires whenever the recorder rolls over to a new file segment, allowing you to update shader uniforms or swap lookup tables in your renderer for dynamic, per-segment visual effects.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}