How Session Trajectory Upload Works in ML Intern for Debugging and Replay

ML Intern captures every agent interaction in a session trajectory and asynchronously uploads it to a Hugging Face dataset via a detached subprocess, ensuring non-blocking persistence for debugging and replay.

The session trajectory upload system in the ML Intern repository provides a robust mechanism for persisting agent execution history without disrupting the main event loop. By serializing interactions to local JSON before asynchronously uploading to versioned Hugging Face datasets, developers gain immutable audit trails and replay capabilities. This architecture leverages session_uploader.py as a standalone upload agent and implements exponential backoff retries to handle network transient failures.

Core Architecture of the Trajectory Upload System

The upload workflow spans three primary components: the Session class managing local state, the detached uploader script handling network operations, and the trigger mechanisms initiating persistence at key lifecycle events.

The Session Class and Local Persistence

In agent/core/session.py, the Session class orchestrates trajectory serialization and upload initiation. The get_trajectory() method (lines 85-94) serializes the complete interaction history—including messages, tool calls, timestamps, and events—into a structured dictionary format.

Once serialized, save_trajectory_local() (lines 96-105) writes this data to session_logs/session_<id>_<timestamp>.json with initial metadata marking upload_status as "pending" and reserving fields for the eventual upload_url. This local write happens synchronously to ensure data durability before any network operations begin.

The save_and_upload_detached() method (lines 55-85) coordinates the full pipeline: it calls save_trajectory_local(), then spawns a detached subprocess running session_uploader.py upload <path> <repo_id>. This subprocess uses subprocess.Popen with start_new_session=True (or platform equivalent) to detach from the parent process, ensuring the agent loop continues executing regardless of upload latency or network conditions.

The Session Uploader Script

The session_uploader.py file (agent/core/session_uploader.py, lines 22-48) functions as a standalone CLI tool that handles the actual file transfer. It performs four critical operations:

1. Idempotency check: Skips files already marked "success" to prevent duplicates
2. Format conversion: Packs the JSON trajectory into a single-line JSONL record
3. Dataset path construction: Targets sessions/YYYY-MM-DD/<session_id>.jsonl within the configured repository
4. Network upload: Calls huggingface_hub.HfApi.upload_file() with exponential backoff for resilience

After completion, the script updates the local JSON file's upload_status field to "success" or "failed" and populates upload_url with the public Hugging Face dataset URL when applicable.

Upload Triggers and Lifecycle Management

ML Intern initiates trajectory persistence at three critical points defined in agent/core/agent_loop.py:

• Shutdown (lines 52-57): Final save triggered via Handlers.shutdown(session) when the agent loop exits normally
• Auto-save (lines 81-84 in session.py): Periodic saves every config.auto_save_interval turns to minimize data loss
• Emergency save (lines 58-66 in agent_loop.py): Fallback persistence if the loop exits unexpectedly or receives termination signals

Additionally, Session.retry_failed_uploads_detached() (lines 87-107 in session.py) scans the session_logs/ directory for files marked "pending" or "failed" and spawns detached retry subprocesses. This runs automatically at agent startup ( invoked from agent_loop.py lines 22-27) to recover from previous session interruptions.

Data Flow from Local JSON to Hugging Face Dataset

Understanding the end-to-end data flow clarifies how ML Intern maintains data integrity while minimizing main thread blocking:

1. Interaction recording: User and agent messages accumulate in ContextManager during the session
2. Trigger activation: Shutdown, auto-save interval, or emergency handler invokes save_and_upload_detached()
3. Local persistence: The system writes a JSON file to session_logs/ with status "pending"
4. Subprocess spawn: A detached process launches session_uploader.py upload <local_path> <repo_id>
5. Dataset upload: The uploader converts the payload to JSONL, creating or updating the repository path sessions/YYYY-MM-DD/<session_id>.jsonl
6. Status reconciliation: The local file updates to "success" with the public URL, or "failed" for later retry

Because the upload is idempotent—creating the repository if missing and overwriting existing paths—developers can safely re-run uploads without creating duplicate trajectory records.

Debugging Failed Uploads and Replaying Sessions

The trajectory upload system provides multiple pathways for debugging execution issues and replaying historical sessions.

Inspecting Local Trajectory Files

When uploads fail due to authentication errors, network partitions, or repository permission issues, the local JSON files in session_logs/ remain intact and contain the full interaction history. Developers can inspect these files directly to debug agent behavior without needing successful network transmission. Each file includes metadata fields showing the upload attempt status and timestamps.

Manual Retry of Failed Uploads

For sessions stuck in "failed" status, ML Intern provides two recovery mechanisms. The command-line interface allows manual retry:

python -m agent.core.session_uploader retry \
    session_logs \
    your-username/ml-intern-sessions

Programmatically, you can invoke the same logic from Python:

from agent.core.session import Session

Session.retry_failed_uploads_detached(
    directory="session_logs",
    repo_id="your-username/ml-intern-sessions",
)

This scans the directory and re-attempts uploads for all non-successful trajectories.

Replaying Sessions from the Dataset

Uploaded trajectories reside in public Hugging Face datasets (configurable via config.session_dataset_repo). Each session occupies a single JSONL line, making loading straightforward:

from datasets import load_dataset

ds = load_dataset("your-username/ml-intern-sessions", split="train")
session = ds.filter(lambda x: x["session_id"] == "session_1234")

The Hugging Face Hub UI also renders these JSONL files for manual inspection, allowing developers to search, filter, and share specific agent execution traces.

Implementation Examples

Automatic Upload on Agent Shutdown

The built-in shutdown handler automatically triggers trajectory upload without requiring manual intervention:

# Inside your agent loop or teardown logic

from agent.core.handlers import Handlers

await Handlers.shutdown(session)

Execution chain: Handlers.shutdown() → session.save_and_upload_detached() → detached subprocess → session_uploader.py.

Manual Upload of Specific Sessions

To upload a trajectory file manually outside the automatic lifecycle:

python -m agent.core.session_uploader upload \
    session_logs/session_abc123_20240424_153210.json \
    your-username/ml-intern-sessions

This reads the local JSON, converts it to JSONL format, and uploads to the specified dataset repository.

Emergency Save Implementation

For signal handlers or exception blocks where immediate persistence is critical:

local_path = session.save_and_upload_detached("my-org/debug-sessions")
if local_path:
    print(f"Trajectory saved locally at {local_path}; upload in progress.")

This method returns the local file path immediately while the upload continues asynchronously in the background.

Summary

• Non-blocking uploads: The save_and_upload_detached() method in agent/core/session.py spawns separate processes to prevent I/O operations from stalling the agent loop
• Local durability: All trajectories write to session_logs/ as JSON before network transmission, ensuring data survives process crashes
• Resilient retries: The session_uploader.py script implements exponential backoff and the retry_failed_uploads_detached() mechanism handles recovery from transient failures
• Idempotent operations: Uploads can safely repeat without creating duplicate entries in the target Hugging Face dataset
• Debuggable format: Trajectories store as single-line JSONL in dated directories (sessions/YYYY-MM-DD/), compatible with standard data science tools and the Hugging Face Hub interface

Frequently Asked Questions

How does ML Intern ensure session uploads don't block the agent?

ML Intern uses a detached subprocess architecture. When save_and_upload_detached() executes, it calls subprocess.Popen to spawn session_uploader.py as a separate process with start_new_session=True. This detaches the upload I/O from the main agent loop, allowing the agent to continue processing while the uploader handles network transmission and Hugging Face Hub API calls in the background.

What happens if a session trajectory upload fails?

Failed uploads retain the local JSON file in session_logs/ with upload_status set to "failed". The system provides two recovery paths: automatic retry via retry_failed_uploads_detached() (which scans for failed uploads at agent startup), or manual retry using the session_uploader.py retry CLI command. Because the upload logic is idempotent, re-attempting uploads will not create duplicate entries in the dataset.

How can I replay a session from an uploaded trajectory?

Uploaded sessions reside in the Hugging Face dataset specified by config.session_dataset_repo. Each trajectory stores as a single JSONL line in the path sessions/YYYY-MM-DD/<session_id>.jsonl. Load these using standard Hugging Face datasets library calls or view them directly in the Hub UI. The JSONL format preserves the complete message history, tool call sequences, and timestamps necessary for exact replay.

Where are the configuration settings for session uploads defined?

Configuration parameters reside in configs/main_agent_config.json and include:

• save_sessions: Boolean flag enabling/disabling trajectory persistence
• session_dataset_repo: Target Hugging Face dataset repository ID (e.g., "username/ml-intern-sessions")
• auto_save_interval: Number of turns between automatic trajectory saves during long-running sessions

These values are accessible at runtime via the configuration object passed to Session initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}