# How to Create Custom Tasks in SIA: Required Directory Structure Guide

> Learn to create custom tasks in SIA by following the required directory structure. Organize your files with data public, data private, and reference subdirectories, plus task.md and optional evaluate.py.

- Repository: [Hexo Labs/sia](https://github.com/hexo-ai/sia)
- Tags: how-to-guide
- Published: 2026-06-12

---

**To create custom tasks in SIA, you must organize your task directory with `data/public/`, `data/private/`, and `reference/` subdirectories, including a mandatory [`task.md`](https://github.com/hexo-ai/sia/blob/main/task.md) file and optionally an [`evaluate.py`](https://github.com/hexo-ai/sia/blob/main/evaluate.py) script that implements the `evaluate(gen_dir: str) -> dict` interface.**

SIA (Self-Improving Agents) from the `hexo-ai/sia` repository executes an automated improvement loop on task directories that follow a strict schema. When you create custom tasks with the required directory structure, the orchestrator can locate the task description, evaluation data, and reference agent template needed to bootstrap the meta-agent.

## Required Directory Structure

SIA expects a specific layout relative to your task root directory. The [`sia/orchestrator.py`](https://github.com/hexo-ai/sia/blob/main/sia/orchestrator.py) file validates this structure before initiating the self-improvement loop.

### Public Data Folder

The `data/public/` directory contains files the target agent may read during generation. This includes the mandatory [`task.md`](https://github.com/hexo-ai/sia/blob/main/task.md) file describing the problem, any training datasets, and optional test data. According to the source code in [`sia/orchestrator.py`](https://github.com/hexo-ai/sia/blob/main/sia/orchestrator.py), the meta-agent parses [`task.md`](https://github.com/hexo-ai/sia/blob/main/task.md) to generate the initial target-agent implementation.

### Private Evaluation Data

The `data/private/` directory holds held-out data used exclusively by the evaluator. Files placed here are never exposed to the LLM during the generation phase. This separation ensures unbiased evaluation of the generated agents.

### Reference Agent Template

The `reference/` directory must contain the **reference agent** ([`reference_target_agent.py`](https://github.com/hexo-ai/sia/blob/main/reference_target_agent.py)) that serves as the seed for the meta-agent. You can copy the template from [`sia/tasks/_shared/reference_target_agent.py`](https://github.com/hexo-ai/sia/blob/main/sia/tasks/_shared/reference_target_agent.py) into this folder. Optionally, include a [`SAMPLE_TASK_DESCRIPTIONS.md`](https://github.com/hexo-ai/sia/blob/main/SAMPLE_TASK_DESCRIPTIONS.md) file to provide the meta-agent with additional context about similar tasks.

### Evaluation Script

While optional, you can include an [`evaluate.py`](https://github.com/hexo-ai/sia/blob/main/evaluate.py) file in `data/public/` that implements the function `evaluate(gen_dir: str) -> dict`. The orchestrator imports and executes this function after each generation, returning metrics that feed into the feedback agent's improvement prompts.

## Creating the Task Skeleton

Follow these steps to initialize a compliant task directory. This structure aligns with the specifications documented in the repository's walkthrough and README.

```bash

# Create the required directory hierarchy

mkdir -p my-task/{data/public,data/private,reference}

# Add the mandatory task description

cp task_description.md my-task/data/public/task.md

# Add public datasets the agent can access

cp train.json my-task/data/public/
cp questions.json my-task/data/public/

# Add private ground-truth data for evaluation

cp answers.json my-task/data/private/

# Copy the reference agent template from the SIA package

cp sia/tasks/_shared/reference_target_agent.py my-task/reference/

# Optional: Add sample task descriptions to guide the meta-agent

cat > my-task/reference/SAMPLE_TASK_DESCRIPTIONS.md <<'EOF'

# Example Tasks

- Summarize technical documentation
- Classify sentiment in product reviews
EOF

```

## Implementing the Evaluation Function

Create a custom [`evaluate.py`](https://github.com/hexo-ai/sia/blob/main/evaluate.py) in `data/public/` to score generated outputs against your private data. The function must accept a string path to the generation directory and return a dictionary of metrics.

```python

# my-task/data/public/evaluate.py

import json
import pathlib

def evaluate(gen_dir: str) -> dict:
    """Score predictions against ground truth.
    
    Args:
        gen_dir: Path to the directory containing generated files
        
    Returns:
        dict: Evaluation metrics passed to the feedback agent
    """
    # Load predictions written by the target agent

    pred_path = pathlib.Path(gen_dir) / "predictions.json"
    preds = json.loads(pred_path.read_text())
    
    # Load ground truth from private data

    truth_path = pathlib.Path(__file__).parents[1] / "private" / "answers.json"
    truths = json.loads(truth_path.read_text())
    
    # Calculate accuracy metric

    correct = sum(p == t for p, t in zip(preds, truths))
    return {"accuracy": correct / len(truths)}

```

Place this file in `my-task/data/public/`. The orchestrator automatically detects and executes it after each generation cycle, feeding the returned metrics into the feedback loop.

## Running Your Custom Task

Once the directory structure matches the required layout, invoke SIA with the path to your task directory:

```bash
sia run --task_dir ./my-task --max_gen 5 --run_id 1

```

The orchestrator creates per-generation run folders (`runs/run_1/gen_1/`, etc.) containing the generated [`target_agent.py`](https://github.com/hexo-ai/sia/blob/main/target_agent.py), execution logs, and improvement notes. The meta-agent iteratively refines the target agent based on the evaluation scores until reaching the specified `--max_gen` limit.

## Summary

- **Directory layout**: Create `data/public/`, `data/private/`, and `reference/` subdirectories in your task root.
- **Mandatory files**: Include [`task.md`](https://github.com/hexo-ai/sia/blob/main/task.md) in `data/public/` describing the problem, and copy [`reference_target_agent.py`](https://github.com/hexo-ai/sia/blob/main/reference_target_agent.py) from `sia/tasks/_shared/` into the `reference/` folder.
- **Evaluation**: Optionally implement `evaluate(gen_dir: str) -> dict` in [`data/public/evaluate.py`](https://github.com/hexo-ai/sia/blob/main/data/public/evaluate.py) to provide automated scoring for the feedback agent.
- **Execution**: Run `sia run --task_dir <path>` to start the self-improving loop on your custom task.

## Frequently Asked Questions

### What happens if I don't include an evaluate.py file?

If [`evaluate.py`](https://github.com/hexo-ai/sia/blob/main/evaluate.py) is missing from `data/public/`, the orchestrator will still run the self-improvement loop, but the feedback agent will receive less specific guidance about the target agent's performance. While the meta-agent can suggest improvements based on execution logs, providing an evaluation function enables data-driven optimization using concrete metrics from your private dataset.

### Can I modify the reference_target_agent.py template?

Yes, you can customize the [`reference_target_agent.py`](https://github.com/hexo-ai/sia/blob/main/reference_target_agent.py) in your task's `reference/` directory. The orchestrator uses this file as the seed for the first generation. However, ensure the template maintains the expected interface (such as a `solve()` or `run()` method) that the generated code will implement, as the meta-agent assumes this structure when creating subsequent iterations.

### How does SIA handle the private data during execution?

Files in `data/private/` are never exposed to the LLM during the generation phase. The [`sia/orchestrator.py`](https://github.com/hexo-ai/sia/blob/main/sia/orchestrator.py) only passes paths to `data/public/` when setting up the target agent's environment. The private data is accessed solely by your [`evaluate.py`](https://github.com/hexo-ai/sia/blob/main/evaluate.py) function after the generation completes, ensuring a clean separation between training/evaluation data and the agent's working context.

### What is the minimum directory structure required to run a task?

The absolute minimum requires three components: [`data/public/task.md`](https://github.com/hexo-ai/sia/blob/main/data/public/task.md) containing the task description, [`reference/reference_target_agent.py`](https://github.com/hexo-ai/sia/blob/main/reference/reference_target_agent.py) copied from [`sia/tasks/_shared/reference_target_agent.py`](https://github.com/hexo-ai/sia/blob/main/sia/tasks/_shared/reference_target_agent.py), and an empty `data/private/` directory (even if unused). Without these elements, the orchestrator will raise errors during the initialization phase.