Integrating CocoIndex with CI/CD Pipelines: A Complete Technical Guide

CocoIndex integrates seamlessly with CI/CD pipelines using its declarative Python API and incremental Rust engine, enabling fast, deterministic data sync tests that automatically skip unchanged content via function memoization.

CocoIndex is a declarative incremental-sync engine from the cocoindex-io/cocoindex repository that combines a Rust core with an async Python frontend. Because the engine runs as standard Python code with no external service dependencies, it fits naturally into existing CI/CD workflows while providing the performance benefits of incremental updates.

Core Architectural Components for CI/CD

Understanding how CocoIndex structures its runtime helps you integrate it effectively into automated build pipelines.

The coco.App Entry Point

The coco.App class, implemented in python/cocoindex/_internal/app.py, serves as the top-level runnable that bundles your pipeline definition and execution arguments. This class instantiates the Rust engine, triggers the async pipeline, and blocks until all target states are synced, making it ideal for CI environments that require deterministic exit codes.

You invoke the pipeline in CI using app.update_blocking(report_to_stdout=True), which runs the full declarative sync and reports status to stdout for capture by CI logs.

Declarative API Primitives

The pipeline declaration relies on functions in python/cocoindex/_internal/api.py:

• coco.mount – Declares a single source-to-target flow.
• coco.mount_each – Iterates over collections (like directory listings) and mounts an indexer for each item.

These primitives work with connector-specific targets defined in files like python/cocoindex/connectors/postgres/_target.py, which exposes mount_table_target async functions for declaring database destinations.

The CI Workflow Infrastructure

The repository provides reusable workflow definitions:

• .github/workflows/CI.yml – Defines the primary entry point for CI, including triggers, path filters, and matrix strategies across Rust, Python, and multiple operating systems.
• .github/workflows/_test.yml – A reusable workflow called by CI.yml that centralizes build steps, Rust caching via Swatinem/rust-cache, Python dependency syncing with uv, and execution of build-test hooks.

Configuring GitHub Actions for CocoIndex Pipelines

A typical CI configuration checks out the repository, sets up the Rust and Python toolchains, synchronizes dependencies, and executes the pipeline against temporary infrastructure.

Step-by-Step Workflow Configuration

The following workflow demonstrates running a CocoIndex pipeline against a temporary PostgreSQL container:

# .github/workflows/ci-pipeline.yml

name: Run CocoIndex Pipeline

on:
  push:
    branches: [ main ]

jobs:
  pipeline:
    runs-on: ubuntu-latest
    env:
      TESTCONTAINERS_RYUK_DISABLED: "true"
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 1

      - name: Install Python
        uses: actions/setup-python@v6
        with:
          python-version: "3.11"

      - name: Install uv
        uses: astral-sh/setup-uv@v7

      - name: Sync dependencies
        run: uv sync --no-dev --group ci

      - name: Start Postgres test container
        run: |
          docker run -d --name pg -e POSTGRES_PASSWORD=pass \
            -e POSTGRES_USER=coco_user -e POSTGRES_DB=coco_test \
            -p 5432:5432 postgres:16-alpine
          for i in {1..10}; do
            pg_isready -h localhost -p 5432 && break || sleep 1
          done

      - name: Run pipeline
        run: python -m pipeline

This configuration mirrors the internal _test.yml structure but isolates the pipeline execution in a dedicated job.

Writing Testable Pipeline Code

CocoIndex pipelines use standard Python functions decorated with @coco.fn, making them testable with pytest and runnable in CI without modification.

Minimal CI-Ready Pipeline Example

import pathlib
import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.text import RecursiveSplitter

@coco.fn(memo=True)
async def index_file(file: localfs.FileLike, table: postgres.TableTarget) -> None:
    """Index a single file into the Postgres table with memoization."""
    for chunk in RecursiveSplitter().split(await file.read_text()):
        table.declare_row(
            text=chunk.text,
            embedding=compute_embedding(chunk.text)  # Your embedding function

        )

@coco.fn
async def main(src_dir: pathlib.Path, pg_dsn: str) -> None:
    """Define the full pipeline flow."""
    table = await postgres.mount_table_target(
        postgres.connect(pg_dsn), 
        table_name="documents"
    )
    table.declare_vector_index(column="embedding")
    
    await coco.mount_each(
        index_file,
        localfs.walk_dir(src_dir).items(),
        table,
    )

if __name__ == "__main__":
    src = pathlib.Path("./docs")
    dsn = "postgresql://coco_user:pass@localhost:5432/coco_test"
    app = coco.App(
        coco.AppConfig(name="ci-demo"), 
        main, 
        src=src, 
        pg_dsn=dsn
    )
    app.update_blocking(report_to_stdout=True)

The @coco.fn(memo=True) decorator ensures that if a file's content and the function's implementation remain unchanged between runs, the engine skips reprocessing. This is crucial for keeping CI execution times low when processing large datasets.

Leveraging Incremental Sync for Fast CI Performance

The incremental nature of CocoIndex means CI can execute the full pipeline on every pull request without the cost of a full re-index. The engine automatically:

1. Skips unchanged source files detected via content hashing.
2. Memoizes pure functions marked with @coco.fn(memo=True).
3. Calculates deltas between declared state and existing target state (tables, directories, or Kafka topics), applying only necessary changes.

This architecture ensures that CI runs complete in seconds even for pipelines managing gigabytes of data, as the engine performs only delta updates.

Cross-Platform CI Testing

The CI.yml workflow defines a matrix that tests the Rust core and Python bindings across Linux, macOS, and Windows. Because the Python frontend uses async/await patterns compatible with asyncio and the Rust core compiles to native binaries for all major platforms, you can verify pipeline behavior consistently across environments.

To add your pipeline to the existing matrix, include a step after the "Run build-test hooks" phase in CI.yml:

- name: Execute example pipeline
  run: |
    python -c "import examples.pipeline_demo as demo; demo.run()"

Summary

• Standard Python execution: CocoIndex runs as python -m pipeline in any CI system supporting Python 3.11+ and Rust compilation, with entry points in python/cocoindex/_internal/app.py.
• Declarative targeting: Use coco.mount_each and connector-specific mount_*_target functions (e.g., postgres.mount_table_target in python/cocoindex/connectors/postgres/_target.py) to define idempotent sync operations.
• CI speed via memoization: The @coco.fn(memo=True) decorator and incremental engine skip unchanged work, preventing costly re-indexing in pull request builds.
• Reusable workflow structure: Reference .github/workflows/_test.yml for patterns using uv dependency sync, Rust caching, and containerized PostgreSQL testing.
• Cross-platform verification: The existing CI.yml matrix ensures pipelines work identically on Linux, macOS, and Windows runners.

Frequently Asked Questions

Can CocoIndex run in GitHub Actions without a dedicated coordination server?

Yes. CocoIndex operates as a library, not a service. The coco.App class in python/cocoindex/_internal/app.py embeds the Rust engine directly in your Python process. You can run app.update_blocking() in any GitHub Actions job that has Python and Rust installed, with no external coordinator required.

How does CocoIndex handle database dependencies like PostgreSQL in CI?

The recommended approach uses Testcontainers or ephemeral Docker services, as shown in the ci-pipeline.yml example. The Python connectors in python/cocoindex/connectors/postgres/_target.py accept standard DSN strings, allowing you to point pipelines at temporary containers spun up specifically for the CI job.

Will CI pipelines re-index all data on every commit?

No. The engine detects unchanged inputs automatically. When you decorate functions with @coco.fn(memo=True), CocoIndex caches function results based on input hashes and code versions. According to the source implementation in python/cocoindex/_internal/api.py, the engine skips memoized functions when their dependencies haven't changed, ensuring CI runs remain fast even for large datasets.

Is the CocoIndex Rust core compatible with Windows CI runners?

Yes. The CI.yml workflow explicitly tests the Rust core (cargo test) and Python bindings on Windows runners alongside Linux and macOS. The async Python frontend uses standard asyncio patterns that function identically across all platforms, ensuring your coco.App-based pipelines run consistently regardless of the CI environment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}