How AntSK Handles File Uploads and Document Processing: A Complete Pipeline Guide

AntSK processes file uploads through a multi-stage pipeline that saves files to disk, validates them via KMService, queues background import tasks through BackgroundTaskBroker, and processes documents using KernelMemory with configurable LLM embeddings and vector database storage.

The AntSK file upload and document processing pipeline is implemented in the aidotnet/antsk repository as a sophisticated workflow that bridges frontend uploads with backend AI document processing. This system handles everything from initial file validation to vector embedding generation, enabling semantic search capabilities across uploaded documents.

File Upload and Storage

The pipeline begins at the API layer with a dedicated endpoint for receiving multipart form data.

In src/AntSK/Controllers/FileController.cs, the UploadFile method handles POST /api/File/UploadFile requests:

// FileController.cs implementation
public async Task<IActionResult> UploadFile(IFormFile file)
{
    // Validation and storage logic
}

The controller performs the following operations:

• Validates the incoming IFormFile for null checks and size constraints
• Constructs a storage path using FileDirOption.DirectoryPath/files as the base directory
• Generates a unique filename using GUID-based naming to prevent collisions
• Writes the file stream to disk asynchronously
• Returns the physical path (relative to uploads) to the caller for downstream processing

Client-Side Validation and Tracking

Before files reach the background processing queue, the frontend validates and tracks them using KMService.

Located in src/AntSK/Domain/Domain/Service/KMService.cs, this service provides three critical methods for the upload workflow:

Validation via BeforeUpload:

public bool BeforeUpload(UploadFileItem file)
{
    // Checks MIME type against allowed extensions
    // Validates file size < 100MB
    return true; // Only returns true for valid files
}

Completion Tracking via OnSingleCompleted:

When the AntDesign Upload component signals completion, this callback adds the file's name and URL to an internal _fileList collection.

Public Access via FileList Property:

public List<UploadFileItem> FileList { get; }

UI pages read this list to display uploaded files and enable the final import confirmation button.

Background Task Queueing

When the user confirms the upload on the KMS Detail page, the system transitions from synchronous file storage to asynchronous background processing.

In src/AntSK/Pages/KmsPage/KmsDetail.razor.cs, the FileHandleOk method iterates over the validated files:

foreach (var item in iKMService.FileList)
{
    await _httpService.PostAsync(
        $"{NavigationManager.BaseUri}api/KMS/ImportKMSTask",
        new ImportKMSTaskDTO
        {
            ImportType = ImportType.File,
            KmsId = KmsId,
            FilePath = item.Url,
            FileName = item.FileName,
            IsQA = _isQa
        });
}

The ImportKMSTaskDTO (defined in src/AntSK/Domain/Domain/Model/ImportKMSTaskReq.cs) encapsulates:

• ImportType: File or URL
• KmsId: Target knowledge base identifier
• FilePath: Physical path from the upload step
• IsQA: Boolean flag enabling QA-specific processing

Document Processing Pipeline

The core processing logic resides in ImportKMSService, orchestrated through the background task infrastructure.

Task Routing

1. KMSController.ImportKMSTask receives the DTO and converts it to a request object
2. BackgroundTaskBroker enqueues the work
3. BackGroundTaskHandler.ExecuteAsync routes to ImportKMSService.ImportKMSTask

KernelMemory Configuration

Before processing begins, KMService.GetMemoryByKMS constructs a KernelMemoryBuilder with provider-specific configurations:

// Pseudocode representing the configuration logic
var memory = new KernelMemoryBuilder()
    .WithTextGenerationByAIType(chatModel)      // OpenAI, Azure, etc.
    .WithTextEmbeddingGenerationByAIType(embed) // BGE, DashScope, etc.
    .WithMemoryDbByVectorDB(vectorDb)           // Postgres, Redis, Qdrant, etc.
    .Build<MemoryServerless>();

Document Import and Embedding Generation

For file imports, ImportKMSService.ImportKMSTask executes:

1. Memory Instance Retrieval: Calls _kMService.GetMemoryByKMS(km.Id) to obtain the configured MemoryServerless instance

2. Optional QA Pipeline: When req.IsQA is true, the pipeline injects additional handlers:

   • TextExtractionHandler for content parsing
   • QAHandler for question-answer pair generation
   • GenerateEmbeddingsHandler for vector creation
   • SaveRecordsHandler for persistence

3. Document Import: Executes _memory.ImportDocumentAsync with a new Document instance:

   await _memory.ImportDocumentAsync(
       new Document(fileid)
           .AddFile(req.FilePath)
           // Additional configuration based on import type
   );

4. Statistics Collection: Queries the memory store using _kMService.GetDocumentByFileID(km.Id, fileid) to count indexed partitions

5. Metadata Persistence: Updates the KMS detail record via _kmsDetails_Repositories.Update(req.KmsDetail) with:

   • Original filename
   • Generated file GUID
   • Data count (partition count)
   • Success status flag

Summary

• FileController.UploadFile in src/AntSK/Controllers/FileController.cs handles initial file storage with GUID-based naming and returns physical paths
• KMService provides client-side validation through BeforeUpload and tracks completed uploads in FileList for UI coordination
• KmsDetail.razor.cs triggers background processing by posting ImportKMSTaskDTO objects to the KMS controller
• BackgroundTaskBroker and BackGroundTaskHandler route tasks asynchronously to prevent UI blocking during document processing
• ImportKMSService.ImportKMSTask orchestrates the core pipeline: building KernelMemory instances with configurable LLM and embedding providers, executing document import with optional QA handlers, generating embeddings, and persisting metadata to the KMS repository

Frequently Asked Questions

How does AntSK validate file uploads before processing?

AntSK validates uploads at two levels. First, the KMService.BeforeUpload method in src/AntSK/Domain/Domain/Service/KMService.cs checks MIME types against allowed extensions and enforces a 100MB size limit on the client side. Second, FileController.UploadFile performs server-side validation of the IFormFile object before writing the stream to disk with a GUID-based filename.

What background task system does AntSK use for document processing?

AntSK uses a custom BackgroundTaskBroker queueing system implemented in src/AntSK/Domain/Domain/Other/BackGroundTaskHandler.cs. When the UI calls KMSController.ImportKMSTask, the controller converts the DTO to a request and enqueues it. The BackGroundTaskHandler.ExecuteAsync method dequeues tasks and routes them to ImportKMSService.ImportKMSTask, allowing document processing to run asynchronously without blocking the web interface.

How does AntSK configure AI models and vector databases for document embeddings?

AntSK dynamically builds KernelMemory configurations through KMService.GetMemoryByKMS in src/AntSK/Domain/Domain/Service/KMService.cs. The method constructs a KernelMemoryBuilder using helper methods like WithTextGenerationByAIType (for LLMs such as OpenAI or Azure OpenAI), WithTextEmbeddingGenerationByAIType (for embeddings like BGE or DashScope), and WithMemoryDbByVectorDB (for vector stores including Postgres, Redis, Qdrant, or Azure AI Search). This configuration is then used by ImportKMSService to generate embeddings and store document partitions.

What happens when the QA mode is enabled during file import?

When ImportKMSTaskDTO.IsQA is set to true, ImportKMSService.ImportKMSTask in src/AntSK/Domain/Domain/Service/ImportKMSService.cs injects additional processing handlers into the KernelMemory pipeline. These include TextExtractionHandler for parsing document content, QAHandler for generating question-answer pairs from the text, GenerateEmbeddingsHandler for creating vector embeddings, and SaveRecordsHandler for persisting the results. This QA-specific pipeline enables the system to build searchable knowledge bases optimized for question-answering scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}