# How to Integrate Needle with CI/CD Pipelines: A Complete Guide

> Integrate Needle with CI CD pipelines effortlessly using Python tooling like pip and pytest. Streamline data generation, fine-tuning, and build archives for seamless automation.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-27

---

**Needle integrates seamlessly with CI/CD pipelines using standard Python tooling—a `pip install`, `pytest` test suite, and CLI commands for data generation, fine-tuning, and building archives.**

Needle is a Python-first agentic tool framework that ships with a full command-line interface. Because it installs as a standard package and exposes all operations through CLI commands, you can automate the entire model lifecycle—testing, data generation, fine-tuning, and packaging—in any CI/CD system. This guide walks through the implementation using Needle's own GitHub Actions workflow as the reference pattern.

## Understanding Needle's CI/CD Architecture

The repository structure separates concerns cleanly: the [[`.github/workflows/release.yaml`](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml)](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml) workflow orchestrates quality gates and releases, while [[`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py)](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) provides the command interface used in automation scripts.

Key components for pipeline integration:

- **Package definition**: [`pyproject.toml`](https://github.com/cactus-compute/needle/blob/main/pyproject.toml) declares dependencies and optional test extras
- **Test runner**: `pytest` with markers to exclude slow tests
- **CLI entry point**: `needle` command exposing subcommands like `finetune`, `build`, `generate-data`
- **Build system**: `python -m build` creates standard wheel and source distributions

## Setting Up the CI Environment

Needle requires Python 3.12 and builds reliably on Ubuntu runners. The release workflow demonstrates the minimal setup:

```yaml
- uses: actions/setup-python@v5
  with:
    python-version: "3.12"

```

Install the package in editable mode with test dependencies:

```bash
pip install -e ".[test]"

```

This pulls the project source from the working directory and installs `pytest` plus other testing tools specified in [`pyproject.toml`](https://github.com/cactus-compute/needle/blob/main/pyproject.toml). See the implementation in [[`.github/workflows/release.yaml`](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml)](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml) lines 34-38.

## Running Automated Tests

The test suite validates core functionality without slow integration tests:

```bash
pytest -q -m "not slow"

```

This command appears in the release workflow's test step ([[`.github/workflows/release.yaml`](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml)](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml) lines 40-43). The `-q` flag reduces output verbosity for CI logs, while `-m "not slow"` skips tests marked with the `slow` decorator—useful for fast feedback on pull requests.

Test files live in the `tests/` directory, which the workflow discovers automatically through pytest's standard collector.

## Automating Model Operations with the Needle CLI

After tests pass, CI pipelines can invoke Needle's CLI subcommands defined in [[`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py)](https://github.com/cactus-compute/needle/blob/main/needle/cli.py). Each subcommand maps to a stage in the model lifecycle:

| Subcommand | Purpose | Typical CI Stage |
|------------|---------|----------------|
| `generate-data` | Create synthetic training examples from tool definitions | Data preparation |
| `finetune` | Train LoRA adapters on custom datasets | Model training |
| `build` | Package checkpoint + adapter into `.cact` archive | Artifact creation |
| `fetch` | Download platform-specific engine binaries | Environment setup |
| `version` | Output version for logging/debugging | Diagnostics |

### Generating Synthetic Training Data

```bash
needle generate-data \
  --tools ./configs/tools.json \
  --num-samples 500 \
  --output ./data/synthetic.jsonl

```

This produces structured training examples without requiring external APIs, making it safe for automated pipelines. The implementation in [[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) handles tool schema parsing and example generation.

### Fine-Tuning LoRA Adapters

```bash
needle finetune ./data/synthetic.jsonl \
  --epochs 10 \
  --lora-rank 16 \
  --out ./outputs/adapter.pkl

```

Fine-tuning runs on CPU or GPU depending on runner availability. The rank parameter controls adapter size—lower values (8-16) train faster and suit CI environments with time constraints. Source: [[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py).

### Building Deployment Archives

```bash
needle build checkpoints/needle2.pkl \
  --lora ./outputs/adapter.pkl \
  --out ./dist/my_needle.cact

```

The `build` command creates a `.cact` file combining base weights and fine-tuned adapters. This archive format is Needle's native deployment artifact, loadable by the runtime engine.

## Complete CI/CD Pipeline Example

This GitHub Actions workflow implements the full Needle lifecycle:

```yaml
name: Needle CI/CD

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      
      - name: Install Needle
        run: pip install -e ".[test]"
      
      - name: Run tests
        run: pytest -q -m "not slow"

  build-model:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      
      - name: Install Needle
        run: pip install -e "."
      
      - name: Generate training data
        run: |
          needle generate-data \
            --tools configs/tools.json \
            --num-samples 200 \
            --output data/train.jsonl
      
      - name: Fine-tune adapter
        run: |
          needle finetune data/train.jsonl \
            --epochs 5 \
            --lora-rank 8 \
            --out outputs/adapter.pkl
      
      - name: Build .cact archive
        run: |
          needle build checkpoints/needle2.pkl \
            --lora outputs/adapter.pkl \
            --out artifacts/custom_needle.cact
      
      - uses: actions/upload-artifact@v4
        with:
          name: needle-model
          path: artifacts/custom_needle.cact

```

## Packaging and Publishing Releases

The release workflow demonstrates PyPI publication after quality gates pass. Key steps from [[`.github/workflows/release.yaml`](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml)](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml):

1. **Version bumping**: Automatic calculation of next semantic version
2. **Source distribution**: `python -m build` creates `dist/needle-X.Y.Z.tar.gz`
3. **Wheel verification**: `twine check dist/*` validates metadata
4. **PyPI publish**: `pypa/gh-action-pypi-publish@release/v1` handles authentication and upload

```yaml
- name: Build distribution
  run: python -m build

- name: Verify distribution
  run: twine check dist/*

- name: Publish to PyPI
  uses: pypa/gh-action-pypi-publish@release/v1
  with:
    skip-existing: true

```

## Key Files for CI/CD Integration

| File | Purpose | Direct Link |
|------|---------|-------------|
| [`.github/workflows/release.yaml`](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml) | Reference implementation of test, build, and publish pipeline | [View source](https://github.com/cactus-compute/needle/blob/main/.github/workflows/release.yaml) |
| [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) | All CLI subcommands for pipeline automation | [View source](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) |
| [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) | Data generation and fine-tuning logic | [View source](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) |
| [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) | Inference runtime for `needle run` | [View source](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) |
| [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) | Quickstart and documentation links | [View source](https://github.com/cactus-compute/needle/blob/main/README.md) |

## Summary

- **Install Needle in CI** with `pip install -e ".[test]"` to get editable source plus test dependencies
- **Run fast tests** using `pytest -q -m "not slow"` to validate changes without slow integration tests
- **Orchestrate model operations** through `needle` CLI commands: `generate-data`, `finetune`, `build`
- **Package artifacts** as `.cact` archives containing base weights plus fine-tuned adapters
- **Publish releases** with standard Python tooling (`build`, `twine`, `gh-action-pypi-publish`) as demonstrated in the release workflow

## Frequently Asked Questions

### Can I run Needle fine-tuning on GitHub Actions free tier runners?

Yes, but use reduced parameters for time limits. Set `--epochs 3` and `--lora-rank 8` to keep training under 10 minutes. GPU runners (GitHub-hosted or self-hosted) substantially accelerate larger training jobs.

### How do I cache Needle dependencies between CI runs?

Use `actions/cache` or `actions/setup-python` with `cache: 'pip'`. The `pip install -e ".[test]"` command benefits from cached wheels of dependencies like `torch` and `transformers`.

### Does Needle require the engine binary during CI testing?

No—the test suite mocked in `tests/` validates logic without the native engine. Use `needle fetch` to download the engine only for integration tests or final artifact validation stages.

### What's the recommended way to version Needle models in CI?

Embed the Git commit SHA in the `.cact` filename: `my_needle-${GITHUB_SHA::8}.cact`. The `needle build` command accepts any output path, so construct versioned paths in your workflow scripts.