How to Develop a Custom Data Source Connector for WeKnora: A Complete Implementation Guide

Implement the datasource.Connector interface in a new package under internal/datasource/connector/, register it via an init() function, and define UI metadata to make your custom data source available in WeKnora.

WeKnora synchronizes external knowledge bases through a plugin-based connector architecture that abstracts REST API integrations behind a common Go interface. Developing a custom data source connector for WeKnora involves implementing the Connector interface defined in internal/datasource/connector.go, handling authentication via thin HTTP clients, and registering your component in the global ConnectorRegistry. This guide provides the exact file paths, method signatures, and code patterns used in the Tencent/WeKnora codebase to build production-ready connectors.

Understanding the Connector Architecture

WeKnora's connector system relies on a central interface contract and a runtime registry. In internal/datasource/connector.go (lines 96-136), the Connector interface defines six required methods that every implementation must satisfy, while the ConnectorMetadataRegistry (lines 38-62) handles UI discovery. Each connector lives in its own package under internal/datasource/connector/<name>/ and typically consists of three files: connector.go for interface implementation, client.go for HTTP wrappers, and types.go for configuration structs.

The architecture separates concerns between credential management—handled via DataSourceConfig in internal/types/datasource.go—and data fetching. For large datasets, connectors may optionally implement StreamingConnector (lines 77-94 in internal/datasource/connector.go) to support checkpoint-driven incremental sync.

Step-by-Step Implementation Guide

Follow these nine steps to build a production-ready connector:

1. Create the package skeleton
   Create a new directory internal/datasource/connector/<myconnector> and add three empty files: connector.go, client.go, and types.go.

2. Define configuration structs
   In types.go, define a config struct that parses the JSONB configuration stored in WeKnora's database. Include a ParseConfig helper similar to Yuque's parseYuqueConfig implementation.

3. Build the HTTP client
   Implement client.go with a constructor that accepts your config struct and returns an *http.Client. Ensure all outbound requests route through WeKnora's SSRF-protected HTTP client from internal/datasource/httpclient.go.

4. Implement the Connector interface
   In connector.go, define a struct that implements all six required methods:

   • Type() string – returns a unique identifier constant
   • Validate(ctx, cfg) – pings a lightweight endpoint to verify credentials
   • ListResources(ctx, cfg, parentID) – returns available resources for selection
   • ResolveResourceAncestors(...) – returns parent IDs for hierarchical sources (empty slice for flat structures)
   • FetchAll(...) – retrieves all items for given resource IDs
   • FetchIncremental(...) – fetches only changed items using a cursor

5. Add streaming support (optional)
   If your source handles large volumes, implement StreamingConnector with FetchStream that emits items via a StreamHandler and persists cursors after each page. Reference the Feishu Wiki connector in internal/datasource/connector/feishu/wiki/connector.go for a working example.

6. Register the connector
   Add an init() function in your connector package that registers the instance:

   func init() {
       _ = datasource.NewConnectorRegistry().Register(NewConnector())
   }

7. Define UI metadata
   Extend ConnectorMetadataRegistry in internal/datasource/connector.go with a new entry mapping your connector type to name, description, auth type, and capabilities like incremental or streaming.

8. Write unit tests
   Create connector_test.go following existing patterns to test validation, resource listing, and cursor handling. Run go test ./... to ensure no regressions in the datasource package.

9. Document the integration
   Add a README entry under docs/connectors/ describing authentication requirements and environment variables, then submit a pull request to the main branch after running go vet and golint.

Complete Implementation Skeleton

Below is a minimal implementation template for connector.go that satisfies the core interface:

package myconnector

import (
    "context"
    "fmt"

    "github.com/Tencent/WeKnora/internal/datasource"
    "github.com/Tencent/WeKnora/internal/types"
)

type Connector struct{}

func NewConnector() *Connector { return &Connector{} }

func (c *Connector) Type() string { return types.ConnectorTypeMy }

func (c *Connector) Validate(ctx context.Context, cfg *types.DataSourceConfig) error {
    mc, err := parseMyConfig(cfg)
    if err != nil {
        return err
    }
    cli := newClient(mc)
    if err := cli.Ping(ctx); err != nil {
        return fmt.Errorf("myservice validation failed: %w", err)
    }
    return nil
}

func (c *Connector) ListResources(ctx context.Context, cfg *types.DataSourceConfig, parentID string) ([]types.Resource, error) {
    if parentID != "" {
        return []types.Resource{}, nil
    }
    mc, _ := parseMyConfig(cfg)
    cli := newClient(mc)
    repos, err := cli.ListRepos(ctx)
    if err != nil {
        return nil, err
    }
    out := make([]types.Resource, len(repos))
    for i, r := range repos {
        out[i] = types.Resource{
            ExternalID: fmt.Sprintf("%d", r.ID),
            Name:       r.Name,
            Type:       "repo",
            URL:        r.WebURL,
        }
    }
    return out, nil
}

func (c *Connector) ResolveResourceAncestors(ctx context.Context, cfg *types.DataSourceConfig, ids []string) ([]string, error) {
    return []string{}, nil
}

func (c *Connector) FetchAll(ctx context.Context, cfg *types.DataSourceConfig, ids []string) ([]types.FetchedItem, error) {
    items, _, err := c.walk(ctx, cfg, ids, nil, false)
    return items, err
}

func (c *Connector) FetchIncremental(ctx context.Context, cfg *types.DataSourceConfig, cursor *types.SyncCursor) ([]types.FetchedItem, *types.SyncCursor, error) {
    return c.walk(ctx, cfg, cursor.ResourceIDs, cursor.MyCursor, true)
}

func (c *Connector) walk(ctx context.Context, cfg *types.DataSourceConfig, ids []string, cursor interface{}, incremental bool) ([]types.FetchedItem, *types.SyncCursor, error) {
    // Implement API pagination, transform results to types.FetchedItem,
    // and return updated cursor. See Yuque connector for reference.
    return nil, nil, nil
}

Replace placeholder functions like parseMyConfig, newClient, and cli.ListRepos with actual implementations mapping to your external API.

Essential Reference Implementations

Study these existing connectors in the Tencent/WeKnora repository to understand different implementation patterns:

• internal/datasource/connector/yuque/connector.go – Full-featured reference with validation, resource listing, and incremental walk patterns
• internal/datasource/connector/notion/connector.go – Demonstrates handling of tree-structured resources where ResolveResourceAncestors returns meaningful data
• internal/datasource/connector/feishu/wiki/connector.go – Shows streaming connector implementation with checkpoint persistence
• internal/datasource/connector.go – Defines the Connector and StreamingConnector interfaces (lines 38-136) and the registration mechanism
• internal/types/datasource.go – Contains DataSourceConfig and SyncCursor structures used across all connectors
• internal/datasource/httpclient.go – SSRF-protected HTTP client wrapper mandatory for all outbound connections

Summary

• Implement the six-method Connector interface defined in internal/datasource/connector.go to satisfy WeKnora's contract
• Organize code into three standard files: types.go for config, client.go for HTTP logic, and connector.go for interface methods
• Register your connector via an init() function using datasource.NewConnectorRegistry().Register()
• Add UI metadata to ConnectorMetadataRegistry to enable discovery in the frontend
• Use internal/datasource/httpclient.go for all outbound requests to ensure SSRF protection
• Study the Yuque and Feishu Wiki implementations as reference patterns for standard and streaming connectors

Frequently Asked Questions

What Go interface must I implement to create a WeKnora data source connector?

You must implement the datasource.Connector interface defined in internal/datasource/connector.go. This requires six methods: Type(), Validate(), ListResources(), ResolveResourceAncestors(), FetchAll(), and FetchIncremental(). For large datasets, you can optionally implement StreamingConnector to support checkpoint-driven incremental sync via the FetchStream method.

Where do I register my custom connector so WeKnora discovers it at runtime?

Add an init() function in your connector's package that calls datasource.NewConnectorRegistry().Register(NewConnector()), as shown in internal/datasource/connector.go (lines 96-136). This pattern ensures your connector loads automatically when WeKnora starts. You must also add metadata to ConnectorMetadataRegistry so the UI can display your connector in the available sources list.

How should I handle authentication credentials in my connector implementation?

Store credentials in a custom config struct defined in types.go that parses the DataSourceConfig JSONB map from the database. Never hardcode secrets. Implement a Validate() method that uses these credentials to ping a lightweight endpoint, confirming connectivity before the sync begins. Use WeKnora's internal/datasource/httpclient.go wrapper for all HTTP requests to ensure SSRF protection and standardized timeouts.

Does WeKnora support incremental sync for large external datasets?

Yes. Implement the FetchIncremental() method to return only changed items using a cursor (typically a timestamp or revision ID), and optionally implement the StreamingConnector interface for memory-efficient processing. Reference the Feishu Wiki connector in internal/datasource/connector/feishu/wiki/connector.go for a production example of checkpoint-driven streaming with cursor persistence between batches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}