# How to Configure LlamaFactory for Local Model Inference in AntSK

> Learn how to configure LlamaFactory for local model inference in AntSK. Expose local models via an OpenAI-compatible API by setting up modelList.json and starting your service.

- Repository: [AIDotNet/antsk](https://github.com/aidotnet/antsk)
- Tags: how-to-guide
- Published: 2026-02-24

---

**AntSK provides a thin wrapper around the LlamaFactory open-source project that exposes local models via an OpenAI-compatible API on port 8000 after configuring [`modelList.json`](https://github.com/aidotnet/antsk/blob/main/modelList.json) and starting the service through the Add Model UI.**

The `aidotnet/antsk` repository ships with a dedicated `LLamaFactoryService` class that manages the Python runtime, environment variables, and process lifecycle required to run large language models locally. By selecting **AI Type → LlamaFactory** in the Add Model page, AntSK automatically routes requests to `http://localhost:8000/` using the standard OpenAI chat completions format.

## Prerequisites and Configuration Files

Before starting the inference server, you must define which models are available and ensure the Python environment is ready.

### Defining Models in modelList.json

AntSK reads supported model definitions from [`src/AntSK.LLamaFactory/modelList.json`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.LLamaFactory/modelList.json). Each entry maps a display name to a ModelScope identifier and specifies the chat template that LlamaFactory should use when launching the API.

```json
[
  {
    "models": {
      "MyLocal-7B-Chat": {
        "DEFAULT": "myorg/my-local-7b",
        "MODELSCOPE": "myorg/my-local-7b"
      }
    },
    "template": "llama2"
  }
]

```

The `template` value (e.g., `llama2`, `qwen`, `chatglm`) determines the conversation formatting applied by the underlying [`api_antsk.py`](https://github.com/aidotnet/antsk/blob/main/api_antsk.py) script.

### Python Dependencies

The LlamaFactory integration requires packages listed in [`src/AntSK.LLamaFactory/requirements.txt`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.LLamaFactory/requirements.txt). The UI exposes a **Pip Install** button that invokes `LLamaFactoryService.PipInstall()` to install the full dependency set. For single-package updates, use the **Pip Install Name** option.

```csharp
// Called from the UI when the user clicks “Pip Install”
await _ILLamaFactoryService.PipInstall();

```

*Method:* [`LLamaFactoryService.PipInstall()`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.Domain/Domain/Service/LLamaFactoryService.cs#L38-L74)

## Environment Setup and Variables

When `StartLLamaFactory()` launches the Python process, it injects specific environment variables into the `ProcessStartInfo`:

- **`CUDA_VISIBLE_DEVICES`** – Defaults to `0` if not already defined, controlling GPU visibility.
- **`API_PORT`** – Hard-coded to `8000`, defining the local HTTP endpoint.
- **`USE_MODELSCOPE_HUB`** – Defaults to `1`, enabling ModelScope model downloads.

These variables ensure the LlamaFactory API server binds to the expected port and utilizes the correct hardware acceleration without manual shell configuration.

## Starting the Local Inference Server

### UI Configuration (Add Model Page)

Navigate to the **Add Model** page and configure the following:

1. Set **AI Type** to `LLamaFactory`. This automatically populates `_aiModel.EndPoint` with `http://localhost:8000/` and sets the model type to `Chat`.
2. Select the **Model Name** that matches the key defined in [`modelList.json`](https://github.com/aidotnet/antsk/blob/main/modelList.json) (e.g., `MyLocal-7B-Chat`).
3. Click **启动服务** (Start Service). This triggers `LLamaFactoryService.StartLLamaFactory(_aiModel.ModelName)`.

```csharp
private void AITypeChange(AIType aiType)
{
    // …
    case AIType.LLamaFactory:
        _aiModel.EndPoint = "http://localhost:8000/";
        _aiModel.AIModelType = AIModelType.Chat;
        break;
    // …
}

```

*File:* [[`AddModel.razor.cs`](https://github.com/aidotnet/antsk/blob/main/AddModel.razor.cs)](https://github.com/aidotnet/antsk/blob/main/src/AntSK/Pages/Setting/AIModel/AddModel.razor.cs)

### Service Initialization Code

The `StartLLamaFactory` method constructs the process arguments and executes the Python entry point:

```csharp
// UI command – Start Service button
await _ILLamaFactoryService.StartLLamaFactory(_aiModel.ModelName);

```

*Method:* [`LLamaFactoryService.StartLLamaFactory()`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.Domain/Domain/Service/LLamaFactoryService.cs#L111-L156)

The underlying command executed is:

```bash
python api_antsk.py --model_name_or_path <ModelScope> --template <template>

```

This runs inside the bundled `llamafactory` folder located in `src/AntSK.LLamaFactory`.

### Persisting the Service State

After starting the service, the UI updates a dictionary entry to remember the running state across sessions:

```csharp
private async Task StartLFService()
{
    if (string.IsNullOrEmpty(_aiModel.ModelName))
    {
        _ = Message.Error("请先选择模型！", 2);
        return;
    }
    llamaFactoryIsStart = true;
    _logModalVisible = true;
    llamaFactoryDic.Value = "true";
    _IDics_Repositories.Update(llamaFactoryDic);
    _ILLamaFactoryService.LogMessageReceived -= CmdLogHandler;
    _ILLamaFactoryService.LogMessageReceived += CmdLogHandler;
    _ILLamaFactoryService.StartLLamaFactory(_aiModel.ModelName);
}

```

*File:* [[`AddModel.razor.cs`](https://github.com/aidotnet/antsk/blob/main/AddModel.razor.cs)](https://github.com/aidotnet/antsk/blob/main/src/AntSK/Pages/Setting/AIModel/AddModel.razor.cs#L31-L46)

The key `LLamaFactoryConstantcs.IsStartKey` tracks whether the local server is active.

## Verifying the OpenAI-Compatible Endpoint

Once the service is running, the LlamaFactory wrapper exposes a standard OpenAI-compatible chat completions API at:

```

http://localhost:8000/v1/chat/completions

```

AntSK routes all chat requests to this endpoint when the model's **AI Type** is set to `LLamaFactory`. You can verify the server is responding by sending a test request:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MyLocal-7B-Chat",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

```

## Summary

- **Configuration file**: Define models in [`src/AntSK.LLamaFactory/modelList.json`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.LLamaFactory/modelList.json) with ModelScope IDs and templates.
- **Dependencies**: Install Python packages via `LLamaFactoryService.PipInstall()` or the UI's **Pip Install** button.
- **Environment**: The service auto-sets `CUDA_VISIBLE_DEVICES`, `API_PORT=8000`, and `USE_MODELSCOPE_HUB`.
- **UI Setup**: Select **AI Type → LlamaFactory** in the Add Model page to auto-configure the `http://localhost:8000/` endpoint.
- **Service Start**: Click **启动服务** to invoke `StartLLamaFactory()`, which runs `python api_antsk.py` with the specified model and template.
- **Persistence**: The running state is stored using `LLamaFactoryConstantcs.IsStartKey` to survive UI refreshes.

## Frequently Asked Questions

### What file format does modelList.json use?

The [`modelList.json`](https://github.com/aidotnet/antsk/blob/main/modelList.json) file uses a JSON array structure where each object contains a `models` dictionary and a `template` string. The `models` dictionary maps display names to objects containing `DEFAULT` and `MODELSCOPE` keys that point to the model repository identifiers. This file is located at [`src/AntSK.LLamaFactory/modelList.json`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.LLamaFactory/modelList.json).

### Which port does the LlamaFactory service use?

The LlamaFactory service hard-codes `API_PORT` to `8000` in the `ProcessStartInfo` environment variables within `LLamaFactoryService.StartLLamaFactory()`. When you select **AI Type → LlamaFactory** in the AntSK UI, the endpoint is automatically set to `http://localhost:8000/`.

### How do I install Python dependencies for LlamaFactory in AntSK?

You can install dependencies by clicking the **Pip Install** button in the Add Model UI, which calls `LLamaFactoryService.PipInstall()`. This method executes `pip install -r requirements.txt` against the bundled [`src/AntSK.LLamaFactory/requirements.txt`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.LLamaFactory/requirements.txt) file. Alternatively, use **Pip Install Name** to install a single specific package.

### Can I use multiple GPUs with LlamaFactory in AntSK?

Yes, multi-GPU support is available through the `CUDA_VISIBLE_DEVICES` environment variable. The `LLamaFactoryService` checks for this variable and defaults to `"0"` if undefined. To utilize multiple GPUs, set `CUDA_VISIBLE_DEVICES` to a comma-separated list (e.g., `"0,1"`) in your system environment before starting the service, or modify the `ProcessStartInfo` in the source code.