# Integrating CocoIndex with CI/CD Pipelines: A Complete Technical Guide

> Integrate CocoIndex with your CI/CD pipelines easily. Leverage its Python API and Rust engine for fast, deterministic data sync tests with automatic skipping of unchanged content. Boost your deployment efficiency.

- Repository: [CocoIndex/cocoindex](https://github.com/cocoindex-io/cocoindex)
- Tags: how-to-guide
- Published: 2026-05-05

---

**CocoIndex integrates seamlessly with CI/CD pipelines using its declarative Python API and incremental Rust engine, enabling fast, deterministic data sync tests that automatically skip unchanged content via function memoization.**

CocoIndex is a declarative incremental-sync engine from the `cocoindex-io/cocoindex` repository that combines a Rust core with an async Python frontend. Because the engine runs as standard Python code with no external service dependencies, it fits naturally into existing CI/CD workflows while providing the performance benefits of incremental updates.

## Core Architectural Components for CI/CD

Understanding how CocoIndex structures its runtime helps you integrate it effectively into automated build pipelines.

### The coco.App Entry Point

The `coco.App` class, implemented in [`python/cocoindex/_internal/app.py`](https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/_internal/app.py), serves as the top-level runnable that bundles your pipeline definition and execution arguments. This class instantiates the Rust engine, triggers the async pipeline, and blocks until all target states are synced, making it ideal for CI environments that require deterministic exit codes.

You invoke the pipeline in CI using `app.update_blocking(report_to_stdout=True)`, which runs the full declarative sync and reports status to stdout for capture by CI logs.

### Declarative API Primitives

The pipeline declaration relies on functions in [`python/cocoindex/_internal/api.py`](https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/_internal/api.py):

- **`coco.mount`** – Declares a single source-to-target flow.
- **`coco.mount_each`** – Iterates over collections (like directory listings) and mounts an indexer for each item.

These primitives work with connector-specific targets defined in files like [`python/cocoindex/connectors/postgres/_target.py`](https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/connectors/postgres/_target.py), which exposes `mount_table_target` async functions for declaring database destinations.

### The CI Workflow Infrastructure

The repository provides reusable workflow definitions:

- **[`.github/workflows/CI.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/.github/workflows/CI.yml)** – Defines the primary entry point for CI, including triggers, path filters, and matrix strategies across Rust, Python, and multiple operating systems.
- **[`.github/workflows/_test.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/.github/workflows/_test.yml)** – A reusable workflow called by [`CI.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/CI.yml) that centralizes build steps, Rust caching via `Swatinem/rust-cache`, Python dependency syncing with `uv`, and execution of build-test hooks.

## Configuring GitHub Actions for CocoIndex Pipelines

A typical CI configuration checks out the repository, sets up the Rust and Python toolchains, synchronizes dependencies, and executes the pipeline against temporary infrastructure.

### Step-by-Step Workflow Configuration

The following workflow demonstrates running a CocoIndex pipeline against a temporary PostgreSQL container:

```yaml

# .github/workflows/ci-pipeline.yml

name: Run CocoIndex Pipeline

on:
  push:
    branches: [ main ]

jobs:
  pipeline:
    runs-on: ubuntu-latest
    env:
      TESTCONTAINERS_RYUK_DISABLED: "true"
    steps:
      - uses: actions/checkout@v6
        with:
          fetch-depth: 1

      - name: Install Python
        uses: actions/setup-python@v6
        with:
          python-version: "3.11"

      - name: Install uv
        uses: astral-sh/setup-uv@v7

      - name: Sync dependencies
        run: uv sync --no-dev --group ci

      - name: Start Postgres test container
        run: |
          docker run -d --name pg -e POSTGRES_PASSWORD=pass \
            -e POSTGRES_USER=coco_user -e POSTGRES_DB=coco_test \
            -p 5432:5432 postgres:16-alpine
          for i in {1..10}; do
            pg_isready -h localhost -p 5432 && break || sleep 1
          done

      - name: Run pipeline
        run: python -m pipeline

```

This configuration mirrors the internal [`_test.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/_test.yml) structure but isolates the pipeline execution in a dedicated job.

## Writing Testable Pipeline Code

CocoIndex pipelines use standard Python functions decorated with `@coco.fn`, making them testable with `pytest` and runnable in CI without modification.

### Minimal CI-Ready Pipeline Example

```python
import pathlib
import cocoindex as coco
from cocoindex.connectors import localfs, postgres
from cocoindex.ops.text import RecursiveSplitter

@coco.fn(memo=True)
async def index_file(file: localfs.FileLike, table: postgres.TableTarget) -> None:
    """Index a single file into the Postgres table with memoization."""
    for chunk in RecursiveSplitter().split(await file.read_text()):
        table.declare_row(
            text=chunk.text,
            embedding=compute_embedding(chunk.text)  # Your embedding function

        )

@coco.fn
async def main(src_dir: pathlib.Path, pg_dsn: str) -> None:
    """Define the full pipeline flow."""
    table = await postgres.mount_table_target(
        postgres.connect(pg_dsn), 
        table_name="documents"
    )
    table.declare_vector_index(column="embedding")
    
    await coco.mount_each(
        index_file,
        localfs.walk_dir(src_dir).items(),
        table,
    )

if __name__ == "__main__":
    src = pathlib.Path("./docs")
    dsn = "postgresql://coco_user:pass@localhost:5432/coco_test"
    app = coco.App(
        coco.AppConfig(name="ci-demo"), 
        main, 
        src=src, 
        pg_dsn=dsn
    )
    app.update_blocking(report_to_stdout=True)

```

The `@coco.fn(memo=True)` decorator ensures that if a file's content and the function's implementation remain unchanged between runs, the engine skips reprocessing. This is crucial for keeping CI execution times low when processing large datasets.

## Leveraging Incremental Sync for Fast CI Performance

The **incremental** nature of CocoIndex means CI can execute the full pipeline on every pull request without the cost of a full re-index. The engine automatically:

1. Skips unchanged source files detected via content hashing.
2. Memoizes pure functions marked with `@coco.fn(memo=True)`.
3. Calculates deltas between declared state and existing target state (tables, directories, or Kafka topics), applying only necessary changes.

This architecture ensures that CI runs complete in seconds even for pipelines managing gigabytes of data, as the engine performs only delta updates.

## Cross-Platform CI Testing

The [`CI.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/CI.yml) workflow defines a matrix that tests the Rust core and Python bindings across Linux, macOS, and Windows. Because the Python frontend uses async/await patterns compatible with `asyncio` and the Rust core compiles to native binaries for all major platforms, you can verify pipeline behavior consistently across environments.

To add your pipeline to the existing matrix, include a step after the "Run build-test hooks" phase in [`CI.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/CI.yml):

```yaml
- name: Execute example pipeline
  run: |
    python -c "import examples.pipeline_demo as demo; demo.run()"

```

## Summary

- **Standard Python execution**: CocoIndex runs as `python -m pipeline` in any CI system supporting Python 3.11+ and Rust compilation, with entry points in [`python/cocoindex/_internal/app.py`](https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/_internal/app.py).
- **Declarative targeting**: Use `coco.mount_each` and connector-specific `mount_*_target` functions (e.g., `postgres.mount_table_target` in [`python/cocoindex/connectors/postgres/_target.py`](https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/connectors/postgres/_target.py)) to define idempotent sync operations.
- **CI speed via memoization**: The `@coco.fn(memo=True)` decorator and incremental engine skip unchanged work, preventing costly re-indexing in pull request builds.
- **Reusable workflow structure**: Reference [`.github/workflows/_test.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/.github/workflows/_test.yml) for patterns using `uv` dependency sync, Rust caching, and containerized PostgreSQL testing.
- **Cross-platform verification**: The existing [`CI.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/CI.yml) matrix ensures pipelines work identically on Linux, macOS, and Windows runners.

## Frequently Asked Questions

### Can CocoIndex run in GitHub Actions without a dedicated coordination server?

Yes. CocoIndex operates as a library, not a service. The `coco.App` class in [`python/cocoindex/_internal/app.py`](https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/_internal/app.py) embeds the Rust engine directly in your Python process. You can run `app.update_blocking()` in any GitHub Actions job that has Python and Rust installed, with no external coordinator required.

### How does CocoIndex handle database dependencies like PostgreSQL in CI?

The recommended approach uses Testcontainers or ephemeral Docker services, as shown in the [`ci-pipeline.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/ci-pipeline.yml) example. The Python connectors in [`python/cocoindex/connectors/postgres/_target.py`](https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/connectors/postgres/_target.py) accept standard DSN strings, allowing you to point pipelines at temporary containers spun up specifically for the CI job.

### Will CI pipelines re-index all data on every commit?

No. The engine detects unchanged inputs automatically. When you decorate functions with `@coco.fn(memo=True)`, CocoIndex caches function results based on input hashes and code versions. According to the source implementation in [`python/cocoindex/_internal/api.py`](https://github.com/cocoindex-io/cocoindex/blob/main/python/cocoindex/_internal/api.py), the engine skips memoized functions when their dependencies haven't changed, ensuring CI runs remain fast even for large datasets.

### Is the CocoIndex Rust core compatible with Windows CI runners?

Yes. The [`CI.yml`](https://github.com/cocoindex-io/cocoindex/blob/main/CI.yml) workflow explicitly tests the Rust core (`cargo test`) and Python bindings on Windows runners alongside Linux and macOS. The async Python frontend uses standard `asyncio` patterns that function identically across all platforms, ensuring your `coco.App`-based pipelines run consistently regardless of the CI environment.