# How to Integrate Custom OCR Capabilities into the AntSK Platform

> Integrate custom OCR into AntSK by implementing IOcrEngine and using WithCustomImageOcr in KMService.cs to index and search image documents.

- Repository: [AIDotNet/antsk](https://github.com/aidotnet/antsk)
- Tags: how-to-guide
- Published: 2026-02-24

---

**You can integrate custom OCR capabilities into the AntSK platform by implementing the `IOcrEngine` interface and registering your implementation via the `WithCustomImageOcr` method in [`KMService.cs`](https://github.com/aidotnet/antsk/blob/main/KMService.cs), enabling image-based documents to be indexed and searched alongside traditional text content.**

The AntSK platform provides a modular architecture for knowledge management that supports both text and image-based document ingestion. While the system ships with a default **PaddleOCR** implementation, many enterprises require **custom OCR capabilities** to leverage proprietary models, cloud-based APIs, or specialized on-premise solutions. This guide demonstrates the exact integration points within the `aidotnet/antsk` repository, showing you how to extend the platform without modifying core indexing logic.

## How AntSK's Default OCR Implementation Works

The platform's default OCR capability resides in the `AntSKOcrEngine` class located at **[`src/AntSK.OCR/AntSKOcrEngine.cs`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.OCR/AntSKOcrEngine.cs)**. This class implements the **`IOcrEngine`** interface from the `Microsoft.KernelMemory.DataFormats` namespace, which requires a single asynchronous method: `ExtractTextFromImageAsync`.

The default implementation utilizes the `Sdcb.OpenVINO.PaddleOCR` NuGet package to perform offline OCR using the ChineseV4 model. When processing an image, the engine lazily downloads model files on first use, decodes the image stream, and returns the extracted text as a string.

The connection between the OCR engine and the indexing pipeline occurs in **[`src/AntSK.Domain/Domain/Service/KMService.cs`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.Domain/Domain/Service/KMService.cs)**. The private `WithOcr` method checks if a knowledge base has OCR enabled via the `IsOCR` flag:

```csharp
private static void WithOcr(IKernelMemoryBuilder memoryBuild, Kmss kms)
{
    if (kms.IsOCR == 1)
    {
        memoryBuild.WithCustomImageOcr(new AntSKOcrEngine());
    }
}

```

When `WithCustomImageOcr` is invoked, KernelMemory registers the supplied `IOcrEngine` implementation. During document ingestion, any image files (JPEG, PNG, TIFF) encountered are automatically passed to the engine's `ExtractTextFromImageAsync` method, and the resulting text is partitioned and indexed alongside traditional document content.

## Implementing the IOcrEngine Interface for Custom OCR Capabilities

To integrate your own OCR solution, you must create a class that implements the **`IOcrEngine`** interface. This interface defines the contract that KernelMemory expects: a single asynchronous method accepting an image stream and returning extracted text.

Create a new file (for example, [`src/AntSK.OCR/MyCustomOcrEngine.cs`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.OCR/MyCustomOcrEngine.cs)) and implement the interface as follows:

```csharp
using Microsoft.KernelMemory.DataFormats;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

public class MyCustomOcrEngine : IOcrEngine
{
    public Task<string> ExtractTextFromImageAsync(Stream imageContent, 
                                                  CancellationToken cancellationToken = default)
    {
        // Convert the incoming stream to the format required by your OCR SDK
        using var memoryStream = new MemoryStream();
        imageContent.CopyTo(memoryStream);
        byte[] imageBytes = memoryStream.ToArray();
        
        // Example: Call your custom OCR service or library
        // string extractedText = await MyOcrSdk.ProcessAsync(imageBytes, cancellationToken);
        
        string extractedText = "Sample text extracted by custom OCR engine";
        return Task.FromResult(extractedText);
    }
}

```

The implementation must return **plain UTF-8 text**. KernelMemory executes this method synchronously during the indexing pipeline, so the operation should complete quickly or utilize proper asynchronous I/O to avoid blocking. You can integrate cloud-based APIs (such as Azure Computer Vision, AWS Textract, or Google Cloud Vision), on-premise Tesseract installations, or proprietary deep learning models using the same pattern.

## Registering Your Custom OCR Engine in KMService

After implementing the `IOcrEngine` interface, you must register your custom implementation within the KernelMemory pipeline. The registration occurs in the **`WithOcr`** method inside **[`src/AntSK.Domain/Domain/Service/KMService.cs`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.Domain/Domain/Service/KMService.cs)**.

Modify the `WithOcr` method to instantiate your custom engine instead of, or alongside, the default `AntSKOcrEngine`:

```csharp
private static void WithOcr(IKernelMemoryBuilder memoryBuild, Kmss kms)
{
    if (kms.IsOCR == 1)
    {
        // Option 1: Replace the default engine entirely
        memoryBuild.WithCustomImageOcr(new MyCustomOcrEngine());
        
        // Option 2: Support multiple engines via configuration
        /*
        switch (kms.OcrProvider)
        {
            case "Paddle":
                memoryBuild.WithCustomImageOcr(new AntSKOcrEngine());
                break;
            case "Custom":
                memoryBuild.WithCustomImageOcr(new MyCustomOcrEngine());
                break;
            default:
                memoryBuild.WithCustomImageOcr(new AntSKOcrEngine());
                break;
        }
        */
    }
}

```

The **`WithCustomImageOcr`** extension method attaches your implementation to the KernelMemory builder. When the platform processes documents for a knowledge base with OCR enabled, it automatically invokes your registered OCR engine when indexing image files.

## Enabling Custom OCR for Specific Knowledge Bases

OCR processing is controlled at the knowledge base level through the **`IsOCR`** flag in the `Kmss` model. To enable your **custom OCR capabilities** for a specific knowledge base, you must set this flag to `1` (or `true`) in the database.

Execute SQL directly to enable OCR for a specific knowledge base:

```sql
UPDATE Kmss SET IsOCR = 1 WHERE Id = 'your-knowledge-base-id';

```

If you extended the schema to support multiple OCR providers as shown in the registration section, include the provider identifier:

```sql
UPDATE Kmss SET IsOCR = 1, OcrProvider = 'Custom' WHERE Id = 'your-knowledge-base-id';

```

Once enabled, the platform automatically detects the `IsOCR` flag during `GetMemoryByKMS` operations. When users upload image files (JPEG, PNG, TIFF, or PDF with embedded images) through the AntSK interface, the platform invokes your custom `ExtractTextFromImageAsync` implementation. The extracted text is partitioned, embedded, and indexed, making it searchable through the standard **Ask** interface alongside traditional text documents.

## Complete Integration Example: Cloud OCR Service

Here is a complete workflow demonstrating how to integrate a cloud-based OCR API into the AntSK platform.

First, implement the engine in [`src/AntSK.OCR/CloudOcrEngine.cs`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.OCR/CloudOcrEngine.cs):

```csharp
using Microsoft.KernelMemory.DataFormats;
using System.Net.Http;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

public class CloudOcrEngine : IOcrEngine
{
    private readonly HttpClient _httpClient;
    private readonly string _apiKey;
    
    public CloudOcrEngine(string apiKey)
    {
        _apiKey = apiKey;
        _httpClient = new HttpClient();
    }
    
    public async Task<string> ExtractTextFromImageAsync(Stream imageContent, 
                                                       CancellationToken cancellationToken = default)
    {
        using var formData = new MultipartFormDataContent();
        formData.Add(new StreamContent(imageContent), "image", "upload.jpg");
        formData.Add(new StringContent(_apiKey), "api_key");
        
        var response = await _httpClient.PostAsync(
            "https://api.cloudocr.example.com/extract", 
            formData, 
            cancellationToken);
            
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
}

```

Next, register the engine in [`src/AntSK.Domain/Domain/Service/KMService.cs`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.Domain/Domain/Service/KMService.cs):

```csharp
private static void WithOcr(IKernelMemoryBuilder memoryBuild, Kmss kms)
{
    if (kms.IsOCR == 1)
    {
        // Use cloud OCR for high-accuracy processing
        memoryBuild.WithCustomImageOcr(new CloudOcrEngine("your-api-key-here"));
    }
}

```

Finally, enable OCR for your knowledge base and upload an image. The platform will automatically extract text using your cloud service and index it for semantic search.

## Summary

- The AntSK platform uses **KernelMemory** to orchestrate OCR processing through the **`IOcrEngine`** interface.
- The default implementation in [`AntSKOcrEngine.cs`](https://github.com/aidotnet/antsk/blob/main/AntSKOcrEngine.cs) leverages **PaddleOCR**, but you can replace it with any custom solution.
- To integrate **custom OCR capabilities**, implement `IOcrEngine`, register your class via `WithCustomImageOcr` in [`KMService.cs`](https://github.com/aidotnet/antsk/blob/main/KMService.cs), and set `IsOCR = 1` in the knowledge base configuration.
- The integration supports cloud APIs, on-premise models, and proprietary OCR engines without modifying core indexing logic.

## Frequently Asked Questions

### What interface must I implement to add custom OCR capabilities to AntSK?

You must implement the **`IOcrEngine`** interface from the `Microsoft.KernelMemory.DataFormats` namespace. This interface requires a single method: `ExtractTextFromImageAsync(Stream imageContent, CancellationToken cancellationToken)`, which must return the extracted text as a string. The method receives the image as a stream and should handle conversion to your OCR SDK's required format internally.

### Where do I register my custom OCR engine in the AntSK codebase?

Register your implementation in the **`WithOcr`** method located in **[`src/AntSK.Domain/Domain/Service/KMService.cs`](https://github.com/aidotnet/antsk/blob/main/src/AntSK.Domain/Domain/Service/KMService.cs)**. Use the `memoryBuild.WithCustomImageOcr(new YourCustomEngine())` extension method to attach your engine to the KernelMemory builder. This registration only occurs when `kms.IsOCR == 1`, ensuring your custom logic is invoked only for knowledge bases explicitly configured for OCR processing.

### Can I use cloud-based OCR APIs like Azure Computer Vision or AWS Textract with AntSK?

Yes. Since you control the implementation of `ExtractTextFromImageAsync`, you can invoke any external API, including **Azure Computer Vision**, **AWS Textract**, **Google Cloud Vision**, or proprietary REST services. Simply use `HttpClient` within your implementation to send the image stream to your chosen endpoint, await the response, and return the extracted text string to KernelMemory for indexing.

### How do I enable OCR processing for a specific knowledge base?

Set the **`IsOCR`** column to `1` (or `true`) in the **`Kmss`** table for the desired knowledge base. You can execute SQL directly: `UPDATE Kmss SET IsOCR = 1 WHERE Id = 'your-kb-id'`. If you implemented multiple provider support, also set the `OcrProvider` column to your custom identifier. Once enabled, the platform automatically routes image uploads through your registered OCR engine during the indexing process.