How to Integrate Needle with CI/CD Pipelines: A Complete Guide

Needle integrates seamlessly with CI/CD pipelines using standard Python tooling—a pip install, pytest test suite, and CLI commands for data generation, fine-tuning, and building archives.

Needle is a Python-first agentic tool framework that ships with a full command-line interface. Because it installs as a standard package and exposes all operations through CLI commands, you can automate the entire model lifecycle—testing, data generation, fine-tuning, and packaging—in any CI/CD system. This guide walks through the implementation using Needle's own GitHub Actions workflow as the reference pattern.

Understanding Needle's CI/CD Architecture

The repository structure separates concerns cleanly: the [.github/workflows/release.yaml](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml) workflow orchestrates quality gates and releases, while [needle/cli.py](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) provides the command interface used in automation scripts.

Key components for pipeline integration:

• Package definition: pyproject.toml declares dependencies and optional test extras
• Test runner: pytest with markers to exclude slow tests
• CLI entry point: needle command exposing subcommands like finetune, build, generate-data
• Build system: python -m build creates standard wheel and source distributions

Setting Up the CI Environment

Needle requires Python 3.12 and builds reliably on Ubuntu runners. The release workflow demonstrates the minimal setup:

- uses: actions/setup-python@v5
  with:
    python-version: "3.12"

Install the package in editable mode with test dependencies:

pip install -e ".[test]"

This pulls the project source from the working directory and installs pytest plus other testing tools specified in pyproject.toml. See the implementation in [.github/workflows/release.yaml](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml) lines 34-38.

Running Automated Tests

The test suite validates core functionality without slow integration tests:

pytest -q -m "not slow"

This command appears in the release workflow's test step ([.github/workflows/release.yaml](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml) lines 40-43). The -q flag reduces output verbosity for CI logs, while -m "not slow" skips tests marked with the slow decorator—useful for fast feedback on pull requests.

Test files live in the tests/ directory, which the workflow discovers automatically through pytest's standard collector.

Automating Model Operations with the Needle CLI

After tests pass, CI pipelines can invoke Needle's CLI subcommands defined in [needle/cli.py](https://github.com/cactus-compute/needle/blob/main/needle/cli.py). Each subcommand maps to a stage in the model lifecycle:

Subcommand	Purpose	Typical CI Stage
generate-data	Create synthetic training examples from tool definitions	Data preparation
finetune	Train LoRA adapters on custom datasets	Model training
build	Package checkpoint + adapter into .cact archive	Artifact creation
fetch	Download platform-specific engine binaries	Environment setup
version	Output version for logging/debugging	Diagnostics

Generating Synthetic Training Data

needle generate-data \
  --tools ./configs/tools.json \
  --num-samples 500 \
  --output ./data/synthetic.jsonl

This produces structured training examples without requiring external APIs, making it safe for automated pipelines. The implementation in [needle/model/finetune.py](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) handles tool schema parsing and example generation.

Fine-Tuning LoRA Adapters

needle finetune ./data/synthetic.jsonl \
  --epochs 10 \
  --lora-rank 16 \
  --out ./outputs/adapter.pkl

Fine-tuning runs on CPU or GPU depending on runner availability. The rank parameter controls adapter size—lower values (8-16) train faster and suit CI environments with time constraints. Source: [needle/model/finetune.py](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py).

Building Deployment Archives

needle build checkpoints/needle2.pkl \
  --lora ./outputs/adapter.pkl \
  --out ./dist/my_needle.cact

The build command creates a .cact file combining base weights and fine-tuned adapters. This archive format is Needle's native deployment artifact, loadable by the runtime engine.

Complete CI/CD Pipeline Example

This GitHub Actions workflow implements the full Needle lifecycle:

name: Needle CI/CD

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      
      - name: Install Needle
        run: pip install -e ".[test]"
      
      - name: Run tests
        run: pytest -q -m "not slow"

  build-model:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      
      - name: Install Needle
        run: pip install -e "."
      
      - name: Generate training data
        run: |
          needle generate-data \
            --tools configs/tools.json \
            --num-samples 200 \
            --output data/train.jsonl
      
      - name: Fine-tune adapter
        run: |
          needle finetune data/train.jsonl \
            --epochs 5 \
            --lora-rank 8 \
            --out outputs/adapter.pkl
      
      - name: Build .cact archive
        run: |
          needle build checkpoints/needle2.pkl \
            --lora outputs/adapter.pkl \
            --out artifacts/custom_needle.cact
      
      - uses: actions/upload-artifact@v4
        with:
          name: needle-model
          path: artifacts/custom_needle.cact

Packaging and Publishing Releases

The release workflow demonstrates PyPI publication after quality gates pass. Key steps from [.github/workflows/release.yaml](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml):

1. Version bumping: Automatic calculation of next semantic version
2. Source distribution: python -m build creates dist/needle-X.Y.Z.tar.gz
3. Wheel verification: twine check dist/* validates metadata
4. PyPI publish: pypa/gh-action-pypi-publish@release/v1 handles authentication and upload

- name: Build distribution
  run: python -m build

- name: Verify distribution
  run: twine check dist/*

- name: Publish to PyPI
  uses: pypa/gh-action-pypi-publish@release/v1
  with:
    skip-existing: true

Key Files for CI/CD Integration

File	Purpose	Direct Link
.github/workflows/release.yaml	Reference implementation of test, build, and publish pipeline	View source
needle/cli.py	All CLI subcommands for pipeline automation	View source
needle/model/finetune.py	Data generation and fine-tuning logic	View source
needle/model/run.py	Inference runtime for needle run	View source
README.md	Quickstart and documentation links	View source

Summary

• Install Needle in CI with pip install -e ".[test]" to get editable source plus test dependencies
• Run fast tests using pytest -q -m "not slow" to validate changes without slow integration tests
• Orchestrate model operations through needle CLI commands: generate-data, finetune, build
• Package artifacts as .cact archives containing base weights plus fine-tuned adapters
• Publish releases with standard Python tooling (build, twine, gh-action-pypi-publish) as demonstrated in the release workflow

Frequently Asked Questions

Can I run Needle fine-tuning on GitHub Actions free tier runners?

Yes, but use reduced parameters for time limits. Set --epochs 3 and --lora-rank 8 to keep training under 10 minutes. GPU runners (GitHub-hosted or self-hosted) substantially accelerate larger training jobs.

How do I cache Needle dependencies between CI runs?

Use actions/cache or actions/setup-python with cache: 'pip'. The pip install -e ".[test]" command benefits from cached wheels of dependencies like torch and transformers.

Does Needle require the engine binary during CI testing?

No—the test suite mocked in tests/ validates logic without the native engine. Use needle fetch to download the engine only for integration tests or final artifact validation stages.

What's the recommended way to version Needle models in CI?

Embed the Git commit SHA in the .cact filename: my_needle-${GITHUB_SHA::8}.cact. The needle build command accepts any output path, so construct versioned paths in your workflow scripts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}