# Where Are the Data Processing Scripts in DeepTutor? Complete Guide to the scripts/ Directory

> Find DeepTutor data processing scripts within the scripts/ directory. This guide details Python utilities for KB migration, user data transfer, embedding generation, and web UI setup.

- Repository: [✨Data Intelligence Lab@HKU✨/DeepTutor](https://github.com/HKUDS/DeepTutor)
- Tags: how-to-guide
- Published: 2026-04-08

---

**The data processing scripts in DeepTutor are located in the top-level `scripts/` directory, containing standalone Python utilities for knowledge base migration, user data transfer, embedding generation, and web UI initialization.**

DeepTutor is an open-source AI tutoring system that manages structured knowledge bases, user workspaces, and LLM-generated embeddings. All data processing scripts in DeepTutor reside in the repository's `scripts/` folder, providing self-contained utilities that operate on the `data/` directory structure and interact with core configuration services.

## The scripts/ Directory: Core Data Processing Utilities

The `scripts/` package contains short, executable Python programs that handle common data-centric tasks before or after tutoring sessions. These files import from `deeptutor/config` and `deeptutor/services` to ensure consistent path resolution and configuration access across the codebase.

The primary data processing scripts include:

- **[`migrate_kb.py`](https://github.com/HKUDS/DeepTutor/blob/main/migrate_kb.py)** – Migrates legacy knowledge base layouts to the current structure
- **[`migrate_user_data.py`](https://github.com/HKUDS/DeepTutor/blob/main/migrate_user_data.py)** – Transfers user-specific data including settings and logs
- **[`test_embedding.py`](https://github.com/HKUDS/DeepTutor/blob/main/test_embedding.py)** – Generates vector embeddings for document collections
- **[`sync_prompts_from_en.py`](https://github.com/HKUDS/DeepTutor/blob/main/sync_prompts_from_en.py)** – Synchronizes English prompt templates from upstream repositories
- **[`start_web.py`](https://github.com/HKUDS/DeepTutor/blob/main/start_web.py)** – Bootstraps the FastAPI web interface while ensuring data directories exist

Additional helper utilities such as [`check_install.py`](https://github.com/HKUDS/DeepTutor/blob/main/check_install.py), [`audit_prompts.py`](https://github.com/HKUDS/DeepTutor/blob/main/audit_prompts.py), [`start_tour.py`](https://github.com/HKUDS/DeepTutor/blob/main/start_tour.py), [`_cli_kit.py`](https://github.com/HKUDS/DeepTutor/blob/main/_cli_kit.py), and [`generate_roster.py`](https://github.com/HKUDS/DeepTutor/blob/main/generate_roster.py) provide environment validation and data preparation functions.

## Key Data Migration Scripts

### migrate_kb.py: Legacy Knowledge Base Migration

Located at [`scripts/migrate_kb.py`](https://github.com/HKUDS/DeepTutor/blob/main/scripts/migrate_kb.py), this utility walks the old `data/knowledge_bases` layout and copies files into the new directory structure. The script updates metadata files during the migration process, ensuring legacy content remains accessible under the current schema.

### migrate_user_data.py: User Data Transfer

The [`scripts/migrate_user_data.py`](https://github.com/HKUDS/DeepTutor/blob/main/scripts/migrate_user_data.py) script handles migration of user-specific information including settings, logs, and workspace content from previous DeepTutor versions. It moves data into the freshly created `data/user` hierarchy while preserving file integrity and configuration history.

## Embedding Generation and Prompt Management

### test_embedding.py: Vector Generation

For generating embeddings, [`scripts/test_embedding.py`](https://github.com/HKUDS/DeepTutor/blob/main/scripts/test_embedding.py) calls the configured LLM embedding provider and stores resulting vectors under `data/knowledge_bases/<kb>/embeddings`. This script utilizes the embedding adapters defined in `deeptutor/services/embedding/` to support OpenAI-compatible APIs.

### sync_prompts_from_en.py: Prompt Synchronization

The [`scripts/sync_prompts_from_en.py`](https://github.com/HKUDS/DeepTutor/blob/main/scripts/sync_prompts_from_en.py) utility pulls the latest prompt templates from upstream English repositories and writes them to `deeptutor/prompts/`. This ensures the tutoring system operates with current, standardized prompt definitions.

## Web Interface and Environment Setup

### start_web.py: Web UI Bootstrap

The [`scripts/start_web.py`](https://github.com/HKUDS/DeepTutor/blob/main/scripts/start_web.py) script launches the FastAPI/WebSocket server that serves the tutoring interface. Before starting the application, it imports `PathService` to guarantee that `data/user/workspace` and `data/user/logs` directories exist, preventing runtime path errors.

### Supporting Utilities

Several additional scripts in the `scripts/` directory support data processing workflows:

- **[`check_install.py`](https://github.com/HKUDS/DeepTutor/blob/main/check_install.py)** – Validates environment dependencies and directory permissions
- **[`audit_prompts.py`](https://github.com/HKUDS/DeepTutor/blob/main/audit_prompts.py)** – Reviews prompt file integrity and structure
- **[`generate_roster.py`](https://github.com/HKUDS/DeepTutor/blob/main/generate_roster.py)** – Creates user or content rosters for batch operations

## Path Resolution and Configuration Services

Beyond standalone scripts, runtime data handling relies on two core modules that provide canonical paths and configuration access.

### path_service.py: Centralized Directory Management

The [`deeptutor/services/path_service.py`](https://github.com/HKUDS/DeepTutor/blob/main/deeptutor/services/path_service.py) module defines the `PathService` class, which exposes properties like `user_data_dir` and `knowledge_bases_dir`. At line 104, the `user_data_dir` property provides the canonical path that every data processing script imports to ensure consistent directory navigation across the application.

### accessors.py: Configuration Access Layer

Located at [`deeptutor/config/accessors.py`](https://github.com/HKUDS/DeepTutor/blob/main/deeptutor/config/accessors.py), this module offers thin wrappers around global configuration values. The `user_data_dir()` accessor at line 16 provides the configuration-backed path resolution used by scripts when initializing file operations or validating environment state.

## Practical Usage Examples

### Running Knowledge Base Migration

Execute the legacy migration from the repository root:

```bash
python scripts/migrate_kb.py \
    --src ./data/knowledge_bases_legacy \
    --dst ./data/knowledge_bases

```

The script loads configuration via [`deeptutor/config/accessors.py`](https://github.com/HKUDS/DeepTutor/blob/main/deeptutor/config/accessors.py) to locate default directories, then walks the source tree to create the new layout with updated metadata.

### Generating Document Embeddings

The following Python code demonstrates using `PathService` and the embedding adapter to process documents:

```python
import pathlib
import numpy as np
from deeptutor.services.path_service import PathService
from deeptutor.services.embedding.adapters.openai_compatible import OpenAIEmbeddingAdapter

# Resolve the target knowledge base directory

kb_path = PathService().knowledge_bases_dir / "my_kb"

# Initialize the OpenAI-compatible adapter

adapter = OpenAIEmbeddingAdapter()

# Load plain-text documents

docs = [p.read_text(encoding="utf-8") for p in kb_path.rglob("*.txt")]

# Generate and store embeddings

embeddings = adapter.embed_documents(docs)
emb_path = kb_path / "embeddings" / "vectors.npy"
emb_path.parent.mkdir(parents=True, exist_ok=True)
np.save(emb_path, np.array(embeddings))
print(f"Saved {len(embeddings)} embeddings to {emb_path}")

```

### Starting the Web Interface

Launch the application while ensuring data directories exist:

```bash
python scripts/start_web.py --host 0.0.0.0 --port 8000

```

## Summary

- The **data processing scripts in DeepTutor** are centralized in the top-level `scripts/` directory
- **[`migrate_kb.py`](https://github.com/HKUDS/DeepTutor/blob/main/migrate_kb.py)** and **[`migrate_user_data.py`](https://github.com/HKUDS/DeepTutor/blob/main/migrate_user_data.py)** handle structural migrations between versions
- **[`test_embedding.py`](https://github.com/HKUDS/DeepTutor/blob/main/test_embedding.py)** generates vector embeddings using the OpenAI-compatible adapter
- **[`deeptutor/services/path_service.py`](https://github.com/HKUDS/DeepTutor/blob/main/deeptutor/services/path_service.py)** provides canonical path resolution via `PathService`
- **[`deeptutor/config/accessors.py`](https://github.com/HKUDS/DeepTutor/blob/main/deeptutor/config/accessors.py)** offers configuration access for all scripts
- Utilities follow a consistent pattern of loading configuration and validating paths before execution

## Frequently Asked Questions

### Where exactly are the data processing scripts located in the DeepTutor repository?

The data processing scripts are located in the `scripts/` directory at the repository root. This folder contains standalone Python files including [`migrate_kb.py`](https://github.com/HKUDS/DeepTutor/blob/main/migrate_kb.py), [`test_embedding.py`](https://github.com/HKUDS/DeepTutor/blob/main/test_embedding.py), and [`start_web.py`](https://github.com/HKUDS/DeepTutor/blob/main/start_web.py) that handle knowledge base operations, embedding generation, and web server initialization.

### How do the scripts know where to find user data and knowledge bases?

All scripts import `PathService` from [`deeptutor/services/path_service.py`](https://github.com/HKUDS/DeepTutor/blob/main/deeptutor/services/path_service.py), which provides properties like `user_data_dir` and `knowledge_bases_dir`. This centralized service ensures consistent path resolution across the application, reading from configuration values exposed through [`deeptutor/config/accessors.py`](https://github.com/HKUDS/DeepTutor/blob/main/deeptutor/config/accessors.py).

### Can I run the embedding generation script independently of the main application?

Yes. [`scripts/test_embedding.py`](https://github.com/HKUDS/DeepTutor/blob/main/scripts/test_embedding.py) is a self-contained utility that imports the embedding adapters from `deeptutor/services/embedding/` and can generate vectors for any document collection in the knowledge base directory without requiring the full web interface to be running.

### What is the purpose of the sync_prompts_from_en.py script?

The [`sync_prompts_from_en.py`](https://github.com/HKUDS/DeepTutor/blob/main/sync_prompts_from_en.py) script pulls the latest English prompt templates from upstream repositories and synchronizes them into `deeptutor/prompts/`. This ensures the tutoring system uses current prompt definitions and maintains consistency with the main English language repository.