How to Integrate Custom OCR Capabilities into the AntSK Platform

You can integrate custom OCR capabilities into the AntSK platform by implementing the IOcrEngine interface and registering your implementation via the WithCustomImageOcr method in KMService.cs, enabling image-based documents to be indexed and searched alongside traditional text content.

The AntSK platform provides a modular architecture for knowledge management that supports both text and image-based document ingestion. While the system ships with a default PaddleOCR implementation, many enterprises require custom OCR capabilities to leverage proprietary models, cloud-based APIs, or specialized on-premise solutions. This guide demonstrates the exact integration points within the aidotnet/antsk repository, showing you how to extend the platform without modifying core indexing logic.

How AntSK's Default OCR Implementation Works

The platform's default OCR capability resides in the AntSKOcrEngine class located at src/AntSK.OCR/AntSKOcrEngine.cs. This class implements the IOcrEngine interface from the Microsoft.KernelMemory.DataFormats namespace, which requires a single asynchronous method: ExtractTextFromImageAsync.

The default implementation utilizes the Sdcb.OpenVINO.PaddleOCR NuGet package to perform offline OCR using the ChineseV4 model. When processing an image, the engine lazily downloads model files on first use, decodes the image stream, and returns the extracted text as a string.

The connection between the OCR engine and the indexing pipeline occurs in src/AntSK.Domain/Domain/Service/KMService.cs. The private WithOcr method checks if a knowledge base has OCR enabled via the IsOCR flag:

private static void WithOcr(IKernelMemoryBuilder memoryBuild, Kmss kms)
{
    if (kms.IsOCR == 1)
    {
        memoryBuild.WithCustomImageOcr(new AntSKOcrEngine());
    }
}

When WithCustomImageOcr is invoked, KernelMemory registers the supplied IOcrEngine implementation. During document ingestion, any image files (JPEG, PNG, TIFF) encountered are automatically passed to the engine's ExtractTextFromImageAsync method, and the resulting text is partitioned and indexed alongside traditional document content.

Implementing the IOcrEngine Interface for Custom OCR Capabilities

To integrate your own OCR solution, you must create a class that implements the IOcrEngine interface. This interface defines the contract that KernelMemory expects: a single asynchronous method accepting an image stream and returning extracted text.

Create a new file (for example, src/AntSK.OCR/MyCustomOcrEngine.cs) and implement the interface as follows:

using Microsoft.KernelMemory.DataFormats;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

public class MyCustomOcrEngine : IOcrEngine
{
    public Task<string> ExtractTextFromImageAsync(Stream imageContent, 
                                                  CancellationToken cancellationToken = default)
    {
        // Convert the incoming stream to the format required by your OCR SDK
        using var memoryStream = new MemoryStream();
        imageContent.CopyTo(memoryStream);
        byte[] imageBytes = memoryStream.ToArray();
        
        // Example: Call your custom OCR service or library
        // string extractedText = await MyOcrSdk.ProcessAsync(imageBytes, cancellationToken);
        
        string extractedText = "Sample text extracted by custom OCR engine";
        return Task.FromResult(extractedText);
    }
}

The implementation must return plain UTF-8 text. KernelMemory executes this method synchronously during the indexing pipeline, so the operation should complete quickly or utilize proper asynchronous I/O to avoid blocking. You can integrate cloud-based APIs (such as Azure Computer Vision, AWS Textract, or Google Cloud Vision), on-premise Tesseract installations, or proprietary deep learning models using the same pattern.

Registering Your Custom OCR Engine in KMService

After implementing the IOcrEngine interface, you must register your custom implementation within the KernelMemory pipeline. The registration occurs in the WithOcr method inside src/AntSK.Domain/Domain/Service/KMService.cs.

Modify the WithOcr method to instantiate your custom engine instead of, or alongside, the default AntSKOcrEngine:

private static void WithOcr(IKernelMemoryBuilder memoryBuild, Kmss kms)
{
    if (kms.IsOCR == 1)
    {
        // Option 1: Replace the default engine entirely
        memoryBuild.WithCustomImageOcr(new MyCustomOcrEngine());
        
        // Option 2: Support multiple engines via configuration
        /*
        switch (kms.OcrProvider)
        {
            case "Paddle":
                memoryBuild.WithCustomImageOcr(new AntSKOcrEngine());
                break;
            case "Custom":
                memoryBuild.WithCustomImageOcr(new MyCustomOcrEngine());
                break;
            default:
                memoryBuild.WithCustomImageOcr(new AntSKOcrEngine());
                break;
        }
        */
    }
}

The WithCustomImageOcr extension method attaches your implementation to the KernelMemory builder. When the platform processes documents for a knowledge base with OCR enabled, it automatically invokes your registered OCR engine when indexing image files.

Enabling Custom OCR for Specific Knowledge Bases

OCR processing is controlled at the knowledge base level through the IsOCR flag in the Kmss model. To enable your custom OCR capabilities for a specific knowledge base, you must set this flag to 1 (or true) in the database.

Execute SQL directly to enable OCR for a specific knowledge base:

UPDATE Kmss SET IsOCR = 1 WHERE Id = 'your-knowledge-base-id';

If you extended the schema to support multiple OCR providers as shown in the registration section, include the provider identifier:

UPDATE Kmss SET IsOCR = 1, OcrProvider = 'Custom' WHERE Id = 'your-knowledge-base-id';

Once enabled, the platform automatically detects the IsOCR flag during GetMemoryByKMS operations. When users upload image files (JPEG, PNG, TIFF, or PDF with embedded images) through the AntSK interface, the platform invokes your custom ExtractTextFromImageAsync implementation. The extracted text is partitioned, embedded, and indexed, making it searchable through the standard Ask interface alongside traditional text documents.

Complete Integration Example: Cloud OCR Service

Here is a complete workflow demonstrating how to integrate a cloud-based OCR API into the AntSK platform.

First, implement the engine in src/AntSK.OCR/CloudOcrEngine.cs:

using Microsoft.KernelMemory.DataFormats;
using System.Net.Http;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

public class CloudOcrEngine : IOcrEngine
{
    private readonly HttpClient _httpClient;
    private readonly string _apiKey;
    
    public CloudOcrEngine(string apiKey)
    {
        _apiKey = apiKey;
        _httpClient = new HttpClient();
    }
    
    public async Task<string> ExtractTextFromImageAsync(Stream imageContent, 
                                                       CancellationToken cancellationToken = default)
    {
        using var formData = new MultipartFormDataContent();
        formData.Add(new StreamContent(imageContent), "image", "upload.jpg");
        formData.Add(new StringContent(_apiKey), "api_key");
        
        var response = await _httpClient.PostAsync(
            "https://api.cloudocr.example.com/extract", 
            formData, 
            cancellationToken);
            
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

Next, register the engine in src/AntSK.Domain/Domain/Service/KMService.cs:

private static void WithOcr(IKernelMemoryBuilder memoryBuild, Kmss kms)
{
    if (kms.IsOCR == 1)
    {
        // Use cloud OCR for high-accuracy processing
        memoryBuild.WithCustomImageOcr(new CloudOcrEngine("your-api-key-here"));
    }
}

Finally, enable OCR for your knowledge base and upload an image. The platform will automatically extract text using your cloud service and index it for semantic search.

Summary

• The AntSK platform uses KernelMemory to orchestrate OCR processing through the IOcrEngine interface.
• The default implementation in AntSKOcrEngine.cs leverages PaddleOCR, but you can replace it with any custom solution.
• To integrate custom OCR capabilities, implement IOcrEngine, register your class via WithCustomImageOcr in KMService.cs, and set IsOCR = 1 in the knowledge base configuration.
• The integration supports cloud APIs, on-premise models, and proprietary OCR engines without modifying core indexing logic.

Frequently Asked Questions

What interface must I implement to add custom OCR capabilities to AntSK?

You must implement the IOcrEngine interface from the Microsoft.KernelMemory.DataFormats namespace. This interface requires a single method: ExtractTextFromImageAsync(Stream imageContent, CancellationToken cancellationToken), which must return the extracted text as a string. The method receives the image as a stream and should handle conversion to your OCR SDK's required format internally.

Where do I register my custom OCR engine in the AntSK codebase?

Register your implementation in the WithOcr method located in src/AntSK.Domain/Domain/Service/KMService.cs. Use the memoryBuild.WithCustomImageOcr(new YourCustomEngine()) extension method to attach your engine to the KernelMemory builder. This registration only occurs when kms.IsOCR == 1, ensuring your custom logic is invoked only for knowledge bases explicitly configured for OCR processing.

Can I use cloud-based OCR APIs like Azure Computer Vision or AWS Textract with AntSK?

Yes. Since you control the implementation of ExtractTextFromImageAsync, you can invoke any external API, including Azure Computer Vision, AWS Textract, Google Cloud Vision, or proprietary REST services. Simply use HttpClient within your implementation to send the image stream to your chosen endpoint, await the response, and return the extracted text string to KernelMemory for indexing.

How do I enable OCR processing for a specific knowledge base?

Set the IsOCR column to 1 (or true) in the Kmss table for the desired knowledge base. You can execute SQL directly: UPDATE Kmss SET IsOCR = 1 WHERE Id = 'your-kb-id'. If you implemented multiple provider support, also set the OcrProvider column to your custom identifier. Once enabled, the platform automatically routes image uploads through your registered OCR engine during the indexing process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}