How to Configure LlamaFactory for Local Model Inference in AntSK

AntSK provides a thin wrapper around the LlamaFactory open-source project that exposes local models via an OpenAI-compatible API on port 8000 after configuring modelList.json and starting the service through the Add Model UI.

The aidotnet/antsk repository ships with a dedicated LLamaFactoryService class that manages the Python runtime, environment variables, and process lifecycle required to run large language models locally. By selecting AI Type → LlamaFactory in the Add Model page, AntSK automatically routes requests to http://localhost:8000/ using the standard OpenAI chat completions format.

Prerequisites and Configuration Files

Before starting the inference server, you must define which models are available and ensure the Python environment is ready.

Defining Models in modelList.json

AntSK reads supported model definitions from src/AntSK.LLamaFactory/modelList.json. Each entry maps a display name to a ModelScope identifier and specifies the chat template that LlamaFactory should use when launching the API.

[
  {
    "models": {
      "MyLocal-7B-Chat": {
        "DEFAULT": "myorg/my-local-7b",
        "MODELSCOPE": "myorg/my-local-7b"
      }
    },
    "template": "llama2"
  }
]

The template value (e.g., llama2, qwen, chatglm) determines the conversation formatting applied by the underlying api_antsk.py script.

Python Dependencies

The LlamaFactory integration requires packages listed in src/AntSK.LLamaFactory/requirements.txt. The UI exposes a Pip Install button that invokes LLamaFactoryService.PipInstall() to install the full dependency set. For single-package updates, use the Pip Install Name option.

// Called from the UI when the user clicks “Pip Install”
await _ILLamaFactoryService.PipInstall();

Method: LLamaFactoryService.PipInstall()

Environment Setup and Variables

When StartLLamaFactory() launches the Python process, it injects specific environment variables into the ProcessStartInfo:

• CUDA_VISIBLE_DEVICES – Defaults to 0 if not already defined, controlling GPU visibility.
• API_PORT – Hard-coded to 8000, defining the local HTTP endpoint.
• USE_MODELSCOPE_HUB – Defaults to 1, enabling ModelScope model downloads.

These variables ensure the LlamaFactory API server binds to the expected port and utilizes the correct hardware acceleration without manual shell configuration.

Starting the Local Inference Server

UI Configuration (Add Model Page)

Navigate to the Add Model page and configure the following:

1. Set AI Type to LLamaFactory. This automatically populates _aiModel.EndPoint with http://localhost:8000/ and sets the model type to Chat.
2. Select the Model Name that matches the key defined in modelList.json (e.g., MyLocal-7B-Chat).
3. Click 启动服务 (Start Service). This triggers LLamaFactoryService.StartLLamaFactory(_aiModel.ModelName).

private void AITypeChange(AIType aiType)
{
    // …
    case AIType.LLamaFactory:
        _aiModel.EndPoint = "http://localhost:8000/";
        _aiModel.AIModelType = AIModelType.Chat;
        break;
    // …
}

File: [AddModel.razor.cs](https://github.com/aidotnet/antsk/blob/main/src/AntSK/Pages/Setting/AIModel/AddModel.razor.cs)

Service Initialization Code

The StartLLamaFactory method constructs the process arguments and executes the Python entry point:

// UI command – Start Service button
await _ILLamaFactoryService.StartLLamaFactory(_aiModel.ModelName);

Method: LLamaFactoryService.StartLLamaFactory()

The underlying command executed is:

python api_antsk.py --model_name_or_path <ModelScope> --template <template>

This runs inside the bundled llamafactory folder located in src/AntSK.LLamaFactory.

Persisting the Service State

After starting the service, the UI updates a dictionary entry to remember the running state across sessions:

private async Task StartLFService()
{
    if (string.IsNullOrEmpty(_aiModel.ModelName))
    {
        _ = Message.Error("请先选择模型！", 2);
        return;
    }
    llamaFactoryIsStart = true;
    _logModalVisible = true;
    llamaFactoryDic.Value = "true";
    _IDics_Repositories.Update(llamaFactoryDic);
    _ILLamaFactoryService.LogMessageReceived -= CmdLogHandler;
    _ILLamaFactoryService.LogMessageReceived += CmdLogHandler;
    _ILLamaFactoryService.StartLLamaFactory(_aiModel.ModelName);
}

File: [AddModel.razor.cs](https://github.com/aidotnet/antsk/blob/main/src/AntSK/Pages/Setting/AIModel/AddModel.razor.cs#L31-L46)

The key LLamaFactoryConstantcs.IsStartKey tracks whether the local server is active.

Verifying the OpenAI-Compatible Endpoint

Once the service is running, the LlamaFactory wrapper exposes a standard OpenAI-compatible chat completions API at:

http://localhost:8000/v1/chat/completions

AntSK routes all chat requests to this endpoint when the model's AI Type is set to LLamaFactory. You can verify the server is responding by sending a test request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MyLocal-7B-Chat",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Summary

• Configuration file: Define models in src/AntSK.LLamaFactory/modelList.json with ModelScope IDs and templates.
• Dependencies: Install Python packages via LLamaFactoryService.PipInstall() or the UI's Pip Install button.
• Environment: The service auto-sets CUDA_VISIBLE_DEVICES, API_PORT=8000, and USE_MODELSCOPE_HUB.
• UI Setup: Select AI Type → LlamaFactory in the Add Model page to auto-configure the http://localhost:8000/ endpoint.
• Service Start: Click 启动服务 to invoke StartLLamaFactory(), which runs python api_antsk.py with the specified model and template.
• Persistence: The running state is stored using LLamaFactoryConstantcs.IsStartKey to survive UI refreshes.

Frequently Asked Questions

What file format does modelList.json use?

The modelList.json file uses a JSON array structure where each object contains a models dictionary and a template string. The models dictionary maps display names to objects containing DEFAULT and MODELSCOPE keys that point to the model repository identifiers. This file is located at src/AntSK.LLamaFactory/modelList.json.

Which port does the LlamaFactory service use?

The LlamaFactory service hard-codes API_PORT to 8000 in the ProcessStartInfo environment variables within LLamaFactoryService.StartLLamaFactory(). When you select AI Type → LlamaFactory in the AntSK UI, the endpoint is automatically set to http://localhost:8000/.

How do I install Python dependencies for LlamaFactory in AntSK?

You can install dependencies by clicking the Pip Install button in the Add Model UI, which calls LLamaFactoryService.PipInstall(). This method executes pip install -r requirements.txt against the bundled src/AntSK.LLamaFactory/requirements.txt file. Alternatively, use Pip Install Name to install a single specific package.

Can I use multiple GPUs with LlamaFactory in AntSK?

Yes, multi-GPU support is available through the CUDA_VISIBLE_DEVICES environment variable. The LLamaFactoryService checks for this variable and defaults to "0" if undefined. To utilize multiple GPUs, set CUDA_VISIBLE_DEVICES to a comma-separated list (e.g., "0,1") in your system environment before starting the service, or modify the ProcessStartInfo in the source code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}