Dolt API for Programmatic Access: gRPC, Go, and SQL Interfaces Explained

Dolt exposes a gRPC-based Remotes API through the remotesrv server that enables programmatic clone, fetch, push, and commit operations via Go clients, SQL stored procedures, or any language supporting protocol buffers.

The Dolt API for programmatic access provides multiple entry points for applications to interact with versioned databases without using the command line. Whether you are building a data pipeline in Go, integrating with existing SQL infrastructure via JDBC, or developing a custom client in Python or Java, Dolt's architecture supports deep programmatic control through its Remotes API, SQL stored procedures, and embeddable Go libraries.

Architecture of the Dolt Remotes API

Dolt implements a gRPC-based Remotes API that mirrors the functionality of the CLI, allowing applications to read tables, write chunks, and manage repository metadata over the network. The system consists of three primary layers: the protocol buffer definitions, the generated client stubs, and the server implementation.

Protocol Buffer Definitions

The API contract is defined in proto/dolt/services/remotesapi/v1alpha1/chunkstore.proto, which specifies the ChunkStoreService interface. This service handles low-level storage operations including chunk upload/download, repository metadata retrieval, and commit creation. The protocol buffer definitions ensure language-agnostic compatibility, enabling clients in any supported language to communicate with Dolt servers.

Server Implementation (remotesrv)

When a Dolt SQL server starts, it optionally launches the remotesrv component on port 50051 (by default). The server bootstrap logic resides in go/utils/remotesrv/main.go, which registers the ChunkStoreService implementation and begins listening for gRPC connections.

// go/utils/remotesrv/main.go
func main() {
    // Parse flags, configure TLS if needed
    lis, _ := net.Listen("tcp", fmt.Sprintf(":%d", port))
    s := grpc.NewServer()
    // Register the ChunkStoreService implementation
    remotesapi.RegisterChunkStoreServiceServer(s, &remotesrv.Server{})
    s.Serve(lis)
}

The concrete implementation of these RPC methods lives in go/libraries/doltcore/remotestorage/chunk_store.go. The DoltChunkStore struct embeds a gRPC client and translates incoming requests into Dolt internal operations, such as reading Noms/Prolly-tree structures and writing new chunks.

// go/libraries/doltcore/remotestorage/chunk_store.go
type DoltChunkStore struct {
    csClient  remotesapi.ChunkStoreServiceClient
    repoId    *remotesapi.RepoId
    metadata  *remotesapi.GetRepoMetadataResponse
}

func (dcs *DoltChunkStore) GetRepoMetadata(ctx context.Context, req *remotesapi.GetRepoMetadataRequest) (*remotesapi.GetRepoMetadataResponse, error) {
    return dcs.csClient.GetRepoMetadata(ctx, req)
}

Using the gRPC Remotes API Directly

For applications requiring fine-grained control over repository operations, the generated Go client provides direct access to the Remotes API. This approach works for any gRPC-capable language, as the protocol buffer definitions in chunkstore.proto generate compatible clients for Python, Java, C++, and others.

Go Client Example

To interact with a Dolt server programmatically, establish a gRPC connection and instantiate the client using remotesapi.NewChunkStoreServiceClient:

package main

import (
    "context"
    "log"
    "time"

    "github.com/dolthub/dolt/go/gen/proto/dolt/services/remotesapi/v1alpha1"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func main() {
    // Connect to the Dolt SQL server with remotesrv enabled
    conn, err := grpc.Dial("localhost:50051", grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        log.Fatalf("Failed to dial: %v", err)
    }
    defer conn.Close()

    client := remotesapi.NewChunkStoreServiceClient(conn)
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    // Retrieve repository metadata to verify connectivity and format version
    meta, err := client.GetRepoMetadata(ctx, &remotesapi.GetRepoMetadataRequest{
        RepoId: &remotesapi.RepoId{Id: "my-repo"},
        ClientRepoFormat: &remotesapi.ClientRepoFormat{
            FormatVersion: "v0.0.0",
        },
    })
    if err != nil {
        log.Fatalf("GetRepoMetadata error: %v", err)
    }
    log.Printf("Repo metadata: %#v", meta)

    // Check which chunks exist on the remote (optimization for push operations)
    hasResp, err := client.HasChunks(ctx, &remotesapi.HasChunksRequest{
        RepoId: meta.RepoId,
        Hashes: [][]byte{
            {0x00, 0x01 /* ... 30 more bytes ... */},
            {0xFF, 0xEE /* ... */},
        },
    })
    if err != nil {
        log.Fatalf("HasChunks error: %v", err)
    }
    log.Printf("Missing chunks: %v", hasResp.MissingHashes)
}

The HasChunks method allows clients to optimize data transfer by determining which chunks already exist on the remote before uploading. For actually transferring data, use GetDownloadLocations for pull operations and GetUploadLocations for push operations, followed by Commit to finalize the repository state.

SQL-Level Programmatic Access

For applications already using SQL infrastructure, Dolt exposes high-level repository operations through stored procedures. This method requires no Go code and works with any SQL client, including JDBC, ODBC, or standard database drivers.

The dolt_clone Stored Procedure

The CALL dolt_clone(...) command, implemented in go/libraries/doltcore/sqle/dprocedures/dolt_clone.go, provides the same functionality as the CLI dolt clone command but accessible via SQL:

-- Clone a remote repository into the current Dolt database
CALL dolt_clone('https://github.com/example/my-dolt-repo', 'main', 'origin', NULL, -1, NULL);

Parameter breakdown:

1. Remote URL: HTTPS, SSH, or direct Dolt server address
2. Branch: Target branch to clone (defaults to master if omitted)
3. Remote name: Local alias for the remote (typically origin)
4. Local directory: NULL clones into current database; specify a path for custom locations
5. Depth: -1 for full history, or positive integer for shallow clone
6. Remote parameters: Map of connection-specific options (e.g., TLS certificate paths)

Internally, this stored procedure invokes CloneDatabaseFromRemote from go/libraries/doltcore/sqle/database_provider.go, which orchestrates the remote resolution, environment creation, and data loading using the same remotesapi client logic as the gRPC examples above.

Embedding Dolt in Go Applications

When building custom tools or extending Dolt's functionality, you can embed the database engine directly within your Go application. This provides the most flexible programmatic access, combining SQL convenience with Go-level control.

Using the Database Provider

The database_provider.go file exposes CloneDatabaseFromRemote, a high-level helper that handles the entire clone workflow:

func (p *DoltDatabaseProvider) CloneDatabaseFromRemote(
    ctx *sql.Context,
    dbName, branch, remoteName, remoteUrl string,
    depth int, remoteParams map[string]string,
) error {
    // 1. Resolve remote via remotesapi
    // 2. Create Dolt environment and load remote data
    // 3. Register cloned database under dbName
}

Executing via SQL Engine

For embedded applications, instantiate a SQL engine and execute stored procedures programmatically:

package main

import (
    "context"
    "log"

    "github.com/dolthub/dolt/go/libraries/doltcore/env"
    "github.com/dolthub/dolt/go/libraries/doltcore/sqle"
)

func main() {
    // Load or create a Dolt environment
    dEnv, err := env.Load(context.Background(), env.GetCurrentUserHomeDir, nil, env.LocalDirDoltDB, "mydb")
    if err != nil {
        log.Fatalf("Environment load failed: %v", err)
    }

    // Create SQL engine with Dolt extensions
    eng := sqle.NewEngine(dEnv)

    // Execute clone via SQL (equivalent to CALL dolt_clone)
    _, err = eng.Query(
        context.Background(),
        "CALL dolt_clone('https://github.com/example/my-dolt-repo', 'main', 'origin', NULL, -1, NULL)",
        sqle.NewDefaultSession(context.Background()),
    )
    if err != nil {
        log.Fatalf("Clone failed: %v", err)
    }
    log.Println("Repository cloned successfully")
}

This approach, utilizing sqle.NewEngine from go/libraries/doltcore/sqle/engine.go, allows seamless integration of versioned data operations into existing Go applications while maintaining compatibility with standard SQL workflows.

Summary

• The Remotes API provides the foundation for all programmatic access, defined in proto/dolt/services/remotesapi/v1alpha1/chunkstore.proto and implemented in remotestorage/chunk_store.go.

• Direct gRPC clients offer language-agnostic access using remotesapi.NewChunkStoreServiceClient, supporting operations like GetRepoMetadata, HasChunks, and Commit for custom synchronization logic.

• SQL stored procedures such as CALL dolt_clone(...) enable immediate integration with existing database infrastructure without requiring Go code, delegating to CloneDatabaseFromRemote in the database provider.

• Embedded Go applications can leverage sqle.NewEngine and the database provider to programmatically manage repositories while maintaining full access to Dolt's versioning capabilities.

Frequently Asked Questions

What port does the Dolt Remotes API use by default?

The remotesrv server listens on port 50051 by default, as implemented in go/utils/remotesrv/main.go. This can be configured via command-line flags when starting the Dolt SQL server with remotes API enabled.

Can I use the Dolt API from languages other than Go?

Yes. Because the API is defined in protocol buffers (chunkstore.proto), you can generate clients for Python, Java, C++, Ruby, and any other language supported by gRPC. The generated stubs communicate with the same ChunkStoreService that Go clients use.

How does CALL dolt_clone differ from the command-line dolt clone?

Both use identical underlying logic through CloneDatabaseFromRemote in database_provider.go. The SQL stored procedure simply provides a programmatic entry point for applications connecting via JDBC, ODBC, or other SQL drivers, whereas the CLI tool is a standalone binary interface.

Is authentication required for the Remotes API?

Authentication depends on the remote server configuration. When connecting to DoltHub or authenticated remotes, the client must provide appropriate credentials through the remoteParams map in CloneDatabaseFromRemote or via connection strings in SQL calls. The gRPC client supports TLS credentials through standard grpc.DialOptions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}