# How to Develop a Custom Data Source Connector for WeKnora: A Complete Implementation Guide

> Learn how to develop a custom data source connector for WeKnora. Implement the datasource Connector interface and register it to integrate your data seamlessly.

- Repository: [Tencent/WeKnora](https://github.com/tencent/WeKnora)
- Tags: how-to-guide
- Published: 2026-09-13

---

**Implement the `datasource.Connector` interface in a new package under `internal/datasource/connector/`, register it via an `init()` function, and define UI metadata to make your custom data source available in WeKnora.**

WeKnora synchronizes external knowledge bases through a plugin-based connector architecture that abstracts REST API integrations behind a common Go interface. Developing a custom data source connector for WeKnora involves implementing the `Connector` interface defined in [`internal/datasource/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector.go), handling authentication via thin HTTP clients, and registering your component in the global `ConnectorRegistry`. This guide provides the exact file paths, method signatures, and code patterns used in the Tencent/WeKnora codebase to build production-ready connectors.

## Understanding the Connector Architecture

WeKnora's connector system relies on a central interface contract and a runtime registry. In [`internal/datasource/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector.go) (lines 96-136), the `Connector` interface defines six required methods that every implementation must satisfy, while the `ConnectorMetadataRegistry` (lines 38-62) handles UI discovery. Each connector lives in its own package under `internal/datasource/connector/<name>/` and typically consists of three files: [`connector.go`](https://github.com/Tencent/WeKnora/blob/main/connector.go) for interface implementation, [`client.go`](https://github.com/Tencent/WeKnora/blob/main/client.go) for HTTP wrappers, and [`types.go`](https://github.com/Tencent/WeKnora/blob/main/types.go) for configuration structs.

The architecture separates concerns between credential management—handled via `DataSourceConfig` in [`internal/types/datasource.go`](https://github.com/Tencent/WeKnora/blob/main/internal/types/datasource.go)—and data fetching. For large datasets, connectors may optionally implement `StreamingConnector` (lines 77-94 in [`internal/datasource/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector.go)) to support checkpoint-driven incremental sync.

## Step-by-Step Implementation Guide

Follow these nine steps to build a production-ready connector:

1. **Create the package skeleton**  
   Create a new directory `internal/datasource/connector/<myconnector>` and add three empty files: [`connector.go`](https://github.com/Tencent/WeKnora/blob/main/connector.go), [`client.go`](https://github.com/Tencent/WeKnora/blob/main/client.go), and [`types.go`](https://github.com/Tencent/WeKnora/blob/main/types.go).

2. **Define configuration structs**  
   In [`types.go`](https://github.com/Tencent/WeKnora/blob/main/types.go), define a config struct that parses the JSONB configuration stored in WeKnora's database. Include a `ParseConfig` helper similar to Yuque's `parseYuqueConfig` implementation.

3. **Build the HTTP client**  
   Implement [`client.go`](https://github.com/Tencent/WeKnora/blob/main/client.go) with a constructor that accepts your config struct and returns an `*http.Client`. Ensure all outbound requests route through WeKnora's SSRF-protected HTTP client from [`internal/datasource/httpclient.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/httpclient.go).

4. **Implement the Connector interface**  
   In [`connector.go`](https://github.com/Tencent/WeKnora/blob/main/connector.go), define a struct that implements all six required methods:
   - `Type() string` – returns a unique identifier constant
   - `Validate(ctx, cfg)` – pings a lightweight endpoint to verify credentials
   - `ListResources(ctx, cfg, parentID)` – returns available resources for selection
   - `ResolveResourceAncestors(...)` – returns parent IDs for hierarchical sources (empty slice for flat structures)
   - `FetchAll(...)` – retrieves all items for given resource IDs
   - `FetchIncremental(...)` – fetches only changed items using a cursor

5. **Add streaming support (optional)**  
   If your source handles large volumes, implement `StreamingConnector` with `FetchStream` that emits items via a `StreamHandler` and persists cursors after each page. Reference the Feishu Wiki connector in [`internal/datasource/connector/feishu/wiki/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector/feishu/wiki/connector.go) for a working example.

6. **Register the connector**  
   Add an `init()` function in your connector package that registers the instance:

   ```go
   func init() {
       _ = datasource.NewConnectorRegistry().Register(NewConnector())
   }
   ```

7. **Define UI metadata**  
   Extend `ConnectorMetadataRegistry` in [`internal/datasource/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector.go) with a new entry mapping your connector type to name, description, auth type, and capabilities like `incremental` or `streaming`.

8. **Write unit tests**  
   Create [`connector_test.go`](https://github.com/Tencent/WeKnora/blob/main/connector_test.go) following existing patterns to test validation, resource listing, and cursor handling. Run `go test ./...` to ensure no regressions in the datasource package.

9. **Document the integration**  
   Add a README entry under `docs/connectors/` describing authentication requirements and environment variables, then submit a pull request to the `main` branch after running `go vet` and `golint`.

## Complete Implementation Skeleton

Below is a minimal implementation template for [`connector.go`](https://github.com/Tencent/WeKnora/blob/main/connector.go) that satisfies the core interface:

```go
package myconnector

import (
    "context"
    "fmt"

    "github.com/Tencent/WeKnora/internal/datasource"
    "github.com/Tencent/WeKnora/internal/types"
)

type Connector struct{}

func NewConnector() *Connector { return &Connector{} }

func (c *Connector) Type() string { return types.ConnectorTypeMy }

func (c *Connector) Validate(ctx context.Context, cfg *types.DataSourceConfig) error {
    mc, err := parseMyConfig(cfg)
    if err != nil {
        return err
    }
    cli := newClient(mc)
    if err := cli.Ping(ctx); err != nil {
        return fmt.Errorf("myservice validation failed: %w", err)
    }
    return nil
}

func (c *Connector) ListResources(ctx context.Context, cfg *types.DataSourceConfig, parentID string) ([]types.Resource, error) {
    if parentID != "" {
        return []types.Resource{}, nil
    }
    mc, _ := parseMyConfig(cfg)
    cli := newClient(mc)
    repos, err := cli.ListRepos(ctx)
    if err != nil {
        return nil, err
    }
    out := make([]types.Resource, len(repos))
    for i, r := range repos {
        out[i] = types.Resource{
            ExternalID: fmt.Sprintf("%d", r.ID),
            Name:       r.Name,
            Type:       "repo",
            URL:        r.WebURL,
        }
    }
    return out, nil
}

func (c *Connector) ResolveResourceAncestors(ctx context.Context, cfg *types.DataSourceConfig, ids []string) ([]string, error) {
    return []string{}, nil
}

func (c *Connector) FetchAll(ctx context.Context, cfg *types.DataSourceConfig, ids []string) ([]types.FetchedItem, error) {
    items, _, err := c.walk(ctx, cfg, ids, nil, false)
    return items, err
}

func (c *Connector) FetchIncremental(ctx context.Context, cfg *types.DataSourceConfig, cursor *types.SyncCursor) ([]types.FetchedItem, *types.SyncCursor, error) {
    return c.walk(ctx, cfg, cursor.ResourceIDs, cursor.MyCursor, true)
}

func (c *Connector) walk(ctx context.Context, cfg *types.DataSourceConfig, ids []string, cursor interface{}, incremental bool) ([]types.FetchedItem, *types.SyncCursor, error) {
    // Implement API pagination, transform results to types.FetchedItem,
    // and return updated cursor. See Yuque connector for reference.
    return nil, nil, nil
}

```

Replace placeholder functions like `parseMyConfig`, `newClient`, and `cli.ListRepos` with actual implementations mapping to your external API.

## Essential Reference Implementations

Study these existing connectors in the Tencent/WeKnora repository to understand different implementation patterns:

- **[`internal/datasource/connector/yuque/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector/yuque/connector.go)** – Full-featured reference with validation, resource listing, and incremental walk patterns
- **[`internal/datasource/connector/notion/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector/notion/connector.go)** – Demonstrates handling of tree-structured resources where `ResolveResourceAncestors` returns meaningful data
- **[`internal/datasource/connector/feishu/wiki/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector/feishu/wiki/connector.go)** – Shows streaming connector implementation with checkpoint persistence
- **[`internal/datasource/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector.go)** – Defines the `Connector` and `StreamingConnector` interfaces (lines 38-136) and the registration mechanism
- **[`internal/types/datasource.go`](https://github.com/Tencent/WeKnora/blob/main/internal/types/datasource.go)** – Contains `DataSourceConfig` and `SyncCursor` structures used across all connectors
- **[`internal/datasource/httpclient.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/httpclient.go)** – SSRF-protected HTTP client wrapper mandatory for all outbound connections

## Summary

- Implement the six-method **Connector interface** defined in [`internal/datasource/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector.go) to satisfy WeKnora's contract
- Organize code into **three standard files**: [`types.go`](https://github.com/Tencent/WeKnora/blob/main/types.go) for config, [`client.go`](https://github.com/Tencent/WeKnora/blob/main/client.go) for HTTP logic, and [`connector.go`](https://github.com/Tencent/WeKnora/blob/main/connector.go) for interface methods
- **Register your connector** via an `init()` function using `datasource.NewConnectorRegistry().Register()`
- Add **UI metadata** to `ConnectorMetadataRegistry` to enable discovery in the frontend
- Use **[`internal/datasource/httpclient.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/httpclient.go)** for all outbound requests to ensure SSRF protection
- Study the **Yuque and Feishu Wiki** implementations as reference patterns for standard and streaming connectors

## Frequently Asked Questions

### What Go interface must I implement to create a WeKnora data source connector?

You must implement the `datasource.Connector` interface defined in [`internal/datasource/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector.go). This requires six methods: `Type()`, `Validate()`, `ListResources()`, `ResolveResourceAncestors()`, `FetchAll()`, and `FetchIncremental()`. For large datasets, you can optionally implement `StreamingConnector` to support checkpoint-driven incremental sync via the `FetchStream` method.

### Where do I register my custom connector so WeKnora discovers it at runtime?

Add an `init()` function in your connector's package that calls `datasource.NewConnectorRegistry().Register(NewConnector())`, as shown in [`internal/datasource/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector.go) (lines 96-136). This pattern ensures your connector loads automatically when WeKnora starts. You must also add metadata to `ConnectorMetadataRegistry` so the UI can display your connector in the available sources list.

### How should I handle authentication credentials in my connector implementation?

Store credentials in a custom config struct defined in [`types.go`](https://github.com/Tencent/WeKnora/blob/main/types.go) that parses the `DataSourceConfig` JSONB map from the database. Never hardcode secrets. Implement a `Validate()` method that uses these credentials to ping a lightweight endpoint, confirming connectivity before the sync begins. Use WeKnora's [`internal/datasource/httpclient.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/httpclient.go) wrapper for all HTTP requests to ensure SSRF protection and standardized timeouts.

### Does WeKnora support incremental sync for large external datasets?

Yes. Implement the `FetchIncremental()` method to return only changed items using a cursor (typically a timestamp or revision ID), and optionally implement the `StreamingConnector` interface for memory-efficient processing. Reference the Feishu Wiki connector in [`internal/datasource/connector/feishu/wiki/connector.go`](https://github.com/Tencent/WeKnora/blob/main/internal/datasource/connector/feishu/wiki/connector.go) for a production example of checkpoint-driven streaming with cursor persistence between batches.