# How Session Trajectory Upload Works in ML Intern for Debugging and Replay

> Learn how ML Intern's session trajectory upload asynchronously saves agent interactions for debugging and replay. Ensure non-blocking persistence with detached subprocesses.

- Repository: [Hugging Face/ml-intern](https://github.com/huggingface/ml-intern)
- Tags: internals
- Published: 2026-04-24

---

**ML Intern captures every agent interaction in a session trajectory and asynchronously uploads it to a Hugging Face dataset via a detached subprocess, ensuring non-blocking persistence for debugging and replay.**

The **session trajectory upload** system in the ML Intern repository provides a robust mechanism for persisting agent execution history without disrupting the main event loop. By serializing interactions to local JSON before asynchronously uploading to versioned Hugging Face datasets, developers gain immutable audit trails and replay capabilities. This architecture leverages [`session_uploader.py`](https://github.com/huggingface/ml-intern/blob/main/session_uploader.py) as a standalone upload agent and implements exponential backoff retries to handle network transient failures.

## Core Architecture of the Trajectory Upload System

The upload workflow spans three primary components: the `Session` class managing local state, the detached uploader script handling network operations, and the trigger mechanisms initiating persistence at key lifecycle events.

### The Session Class and Local Persistence

In [`agent/core/session.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/session.py), the `Session` class orchestrates trajectory serialization and upload initiation. The **`get_trajectory()`** method (lines 85-94) serializes the complete interaction history—including messages, tool calls, timestamps, and events—into a structured dictionary format.

Once serialized, **`save_trajectory_local()`** (lines 96-105) writes this data to `session_logs/session_<id>_<timestamp>.json` with initial metadata marking `upload_status` as `"pending"` and reserving fields for the eventual `upload_url`. This local write happens synchronously to ensure data durability before any network operations begin.

The **`save_and_upload_detached()`** method (lines 55-85) coordinates the full pipeline: it calls `save_trajectory_local()`, then spawns a detached subprocess running `session_uploader.py upload <path> <repo_id>`. This subprocess uses `subprocess.Popen` with `start_new_session=True` (or platform equivalent) to detach from the parent process, ensuring the agent loop continues executing regardless of upload latency or network conditions.

### The Session Uploader Script

The **[`session_uploader.py`](https://github.com/huggingface/ml-intern/blob/main/session_uploader.py)** file ([`agent/core/session_uploader.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/session_uploader.py), lines 22-48) functions as a standalone CLI tool that handles the actual file transfer. It performs four critical operations:

1. **Idempotency check**: Skips files already marked `"success"` to prevent duplicates
2. **Format conversion**: Packs the JSON trajectory into a single-line JSONL record
3. **Dataset path construction**: Targets `sessions/YYYY-MM-DD/<session_id>.jsonl` within the configured repository
4. **Network upload**: Calls `huggingface_hub.HfApi.upload_file()` with exponential backoff for resilience

After completion, the script updates the local JSON file's `upload_status` field to `"success"` or `"failed"` and populates `upload_url` with the public Hugging Face dataset URL when applicable.

### Upload Triggers and Lifecycle Management

ML Intern initiates trajectory persistence at three critical points defined in [`agent/core/agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/agent_loop.py):

- **Shutdown** (lines 52-57): Final save triggered via `Handlers.shutdown(session)` when the agent loop exits normally
- **Auto-save** (lines 81-84 in [`session.py`](https://github.com/huggingface/ml-intern/blob/main/session.py)): Periodic saves every `config.auto_save_interval` turns to minimize data loss
- **Emergency save** (lines 58-66 in [`agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent_loop.py)): Fallback persistence if the loop exits unexpectedly or receives termination signals

Additionally, **`Session.retry_failed_uploads_detached()`** (lines 87-107 in [`session.py`](https://github.com/huggingface/ml-intern/blob/main/session.py)) scans the `session_logs/` directory for files marked `"pending"` or `"failed"` and spawns detached retry subprocesses. This runs automatically at agent startup ( invoked from [`agent_loop.py`](https://github.com/huggingface/ml-intern/blob/main/agent_loop.py) lines 22-27) to recover from previous session interruptions.

## Data Flow from Local JSON to Hugging Face Dataset

Understanding the end-to-end data flow clarifies how ML Intern maintains data integrity while minimizing main thread blocking:

1. **Interaction recording**: User and agent messages accumulate in `ContextManager` during the session
2. **Trigger activation**: Shutdown, auto-save interval, or emergency handler invokes `save_and_upload_detached()`
3. **Local persistence**: The system writes a JSON file to `session_logs/` with status `"pending"`
4. **Subprocess spawn**: A detached process launches `session_uploader.py upload <local_path> <repo_id>`
5. **Dataset upload**: The uploader converts the payload to JSONL, creating or updating the repository path `sessions/YYYY-MM-DD/<session_id>.jsonl`
6. **Status reconciliation**: The local file updates to `"success"` with the public URL, or `"failed"` for later retry

Because the upload is **idempotent**—creating the repository if missing and overwriting existing paths—developers can safely re-run uploads without creating duplicate trajectory records.

## Debugging Failed Uploads and Replaying Sessions

The trajectory upload system provides multiple pathways for debugging execution issues and replaying historical sessions.

### Inspecting Local Trajectory Files

When uploads fail due to authentication errors, network partitions, or repository permission issues, the local JSON files in `session_logs/` remain intact and contain the full interaction history. Developers can inspect these files directly to debug agent behavior without needing successful network transmission. Each file includes metadata fields showing the upload attempt status and timestamps.

### Manual Retry of Failed Uploads

For sessions stuck in `"failed"` status, ML Intern provides two recovery mechanisms. The command-line interface allows manual retry:

```bash
python -m agent.core.session_uploader retry \
    session_logs \
    your-username/ml-intern-sessions

```

Programmatically, you can invoke the same logic from Python:

```python
from agent.core.session import Session

Session.retry_failed_uploads_detached(
    directory="session_logs",
    repo_id="your-username/ml-intern-sessions",
)

```

This scans the directory and re-attempts uploads for all non-successful trajectories.

### Replaying Sessions from the Dataset

Uploaded trajectories reside in public Hugging Face datasets (configurable via `config.session_dataset_repo`). Each session occupies a single JSONL line, making loading straightforward:

```python
from datasets import load_dataset

ds = load_dataset("your-username/ml-intern-sessions", split="train")
session = ds.filter(lambda x: x["session_id"] == "session_1234")

```

The Hugging Face Hub UI also renders these JSONL files for manual inspection, allowing developers to search, filter, and share specific agent execution traces.

## Implementation Examples

### Automatic Upload on Agent Shutdown

The built-in shutdown handler automatically triggers trajectory upload without requiring manual intervention:

```python

# Inside your agent loop or teardown logic

from agent.core.handlers import Handlers

await Handlers.shutdown(session)

```

*Execution chain*: `Handlers.shutdown()` → `session.save_and_upload_detached()` → detached subprocess → [`session_uploader.py`](https://github.com/huggingface/ml-intern/blob/main/session_uploader.py).

### Manual Upload of Specific Sessions

To upload a trajectory file manually outside the automatic lifecycle:

```bash
python -m agent.core.session_uploader upload \
    session_logs/session_abc123_20240424_153210.json \
    your-username/ml-intern-sessions

```

This reads the local JSON, converts it to JSONL format, and uploads to the specified dataset repository.

### Emergency Save Implementation

For signal handlers or exception blocks where immediate persistence is critical:

```python
local_path = session.save_and_upload_detached("my-org/debug-sessions")
if local_path:
    print(f"Trajectory saved locally at {local_path}; upload in progress.")

```

This method returns the local file path immediately while the upload continues asynchronously in the background.

## Summary

- **Non-blocking uploads**: The `save_and_upload_detached()` method in [`agent/core/session.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/session.py) spawns separate processes to prevent I/O operations from stalling the agent loop
- **Local durability**: All trajectories write to `session_logs/` as JSON before network transmission, ensuring data survives process crashes
- **Resilient retries**: The [`session_uploader.py`](https://github.com/huggingface/ml-intern/blob/main/session_uploader.py) script implements exponential backoff and the `retry_failed_uploads_detached()` mechanism handles recovery from transient failures
- **Idempotent operations**: Uploads can safely repeat without creating duplicate entries in the target Hugging Face dataset
- **Debuggable format**: Trajectories store as single-line JSONL in dated directories (`sessions/YYYY-MM-DD/`), compatible with standard data science tools and the Hugging Face Hub interface

## Frequently Asked Questions

### How does ML Intern ensure session uploads don't block the agent?

ML Intern uses a **detached subprocess architecture**. When `save_and_upload_detached()` executes, it calls `subprocess.Popen` to spawn [`session_uploader.py`](https://github.com/huggingface/ml-intern/blob/main/session_uploader.py) as a separate process with `start_new_session=True`. This detaches the upload I/O from the main agent loop, allowing the agent to continue processing while the uploader handles network transmission and Hugging Face Hub API calls in the background.

### What happens if a session trajectory upload fails?

Failed uploads retain the local JSON file in `session_logs/` with `upload_status` set to `"failed"`. The system provides two recovery paths: automatic retry via `retry_failed_uploads_detached()` (which scans for failed uploads at agent startup), or manual retry using the `session_uploader.py retry` CLI command. Because the upload logic is idempotent, re-attempting uploads will not create duplicate entries in the dataset.

### How can I replay a session from an uploaded trajectory?

Uploaded sessions reside in the Hugging Face dataset specified by `config.session_dataset_repo`. Each trajectory stores as a single JSONL line in the path `sessions/YYYY-MM-DD/<session_id>.jsonl`. Load these using standard Hugging Face `datasets` library calls or view them directly in the Hub UI. The JSONL format preserves the complete message history, tool call sequences, and timestamps necessary for exact replay.

### Where are the configuration settings for session uploads defined?

Configuration parameters reside in [`configs/main_agent_config.json`](https://github.com/huggingface/ml-intern/blob/main/configs/main_agent_config.json) and include:
- **`save_sessions`**: Boolean flag enabling/disabling trajectory persistence
- **`session_dataset_repo`**: Target Hugging Face dataset repository ID (e.g., `"username/ml-intern-sessions"`)
- **`auto_save_interval`**: Number of turns between automatic trajectory saves during long-running sessions

These values are accessible at runtime via the configuration object passed to `Session` initialization.