How to Add a Document to Hister Using the API: A Complete Guide

You can add documents to Hister via POST requests to /api/add for single documents or /api/batch for bulk operations, using either the Go client library or raw HTTP calls with JSON or form-encoded payloads.

Hister is an open-source search engine that exposes a straightforward HTTP API for document ingestion. Whether you are building a crawler or integrating existing content sources, adding a document to Hister requires sending structured data to specific REST endpoints implemented in the Go-based server.

API Endpoints for Document Ingestion

Hister provides two primary HTTP endpoints for inserting documents into the index:

• POST /api/add – Accepts a single document via application/json or application/x-www-form-urlencoded content types.
• POST /api/batch – Accepts multiple documents in a single request using a JSON body with an "ops" array, supporting a maximum of 100 operations per call.

Both endpoints enforce request size limits and validate payloads before indexing. The server-side logic resides in server/endpoints.go, where the serveAdd function (lines 731‑800) handles single-document requests and orchestrates the validation and storage pipeline.

Adding a Single Document

Using Raw HTTP Requests

For simple integrations, send a JSON payload directly to the /api/add endpoint:

curl -X POST http://localhost:8080/api/add \
     -H "Content-Type: application/json" \
     -d '{
           "url":"https://example.com/article",
           "title":"Example Article",
           "text":"The article body…"
         }'

Alternatively, use URL-encoded form data if your client does not support JSON:

curl -X POST http://localhost:8080/api/add \
     -H "Content-Type: application/x-www-form-urlencoded" \
     --data-urlencode "url=https://example.com/article" \
     --data-urlencode "title=Example Article" \
     --data-urlencode "text=The article body…"

Using the Go Client Library

The recommended approach for Go projects uses the client package. The AddDocumentJSON method constructs the request and handles response parsing:

import (
    "github.com/asciimoo/hister/client"
    "github.com/asciimoo/hister/server/document"
)

func addSingleDocument(c *client.Client) error {
    doc := &document.Document{
        URL:   "https://example.com/article",
        Title: "Example Article",
        Text:  "The article body…",
    }
    // POST /api/add
    return c.AddDocumentJSON(doc)
}

The client automatically manages connection pooling and error handling while targeting the same underlying server endpoint.

Adding Multiple Documents in Bulk

For high-throughput scenarios, batching documents reduces network overhead. The /api/batch endpoint accepts an "ops" array where each element specifies an "add" operation with a nested document object.

Batch HTTP Request Example

curl -X POST http://localhost:8080/api/batch \
     -H "Content-Type: application/json" \
     -d '{
           "ops": [
             {"op":"add","document":{"url":"https://example.com/first","title":"First","text":"First body"}},
             {"op":"add","document":{"url":"https://example.com/second","title":"Second","text":"Second body"}}
           ]
         }'

The server enforces a hard limit of 100 operations per request and validates the total request size before processing.

Bulk Insertion with the Go Client

The AddDocumentsJSON function in client/document.go (lines 32‑73) abstracts batch construction and automatically handles the 100-item limit by splitting large slices into multiple API requests:

func addMultipleDocuments(c *client.Client) error {
    docs := []*document.Document{
        {URL: "https://example.com/first", Title: "First", Text: "First body"},
        {URL: "https://example.com/second", Title: "Second", Text: "Second body"},
    }
    results, err := c.AddDocumentsJSON(docs)
    if err != nil {
        return err
    }
    for _, r := range results {
        if r.Status != http.StatusCreated {
            return fmt.Errorf("failed: %s (status %d)", r.Error, r.Status)
        }
    }
    return nil
}

This method returns per-document status codes, allowing you to identify specific failures within a batch.

Validation and Response Codes

Before indexing, Hister validates every document using the validateAddDocument function in server/endpoints.go (lines 803‑843). This validation ensures:

• The URL field is well-formed and present.
• For remote-file types, the URL uses an allowed scheme (HTTP/HTTPS).

After validation, the server invokes c.Indexer.AddContext to store the document. The API returns distinct HTTP status codes:

• HTTP 201 Created – The document was successfully indexed.
• HTTP 406 Not Acceptable – The document was rejected due to user-defined indexing rules (skip filters) or validation failures.

Key Implementation Files

Understanding the source structure helps when debugging or extending the API:

• server/endpoints.go – Contains serveAdd (single document handler) and validateAddDocument (validation logic).
• client/document.go – Implements AddDocumentJSON and AddDocumentsJSON for client-side batching.
• server/document/document.go – Defines the Document struct and processing helpers used by both server and client.

Summary

• Hister exposes POST /api/add for single documents and POST /api/batch for bulk operations up to 100 items.
• The API accepts application/json or application/x-www-form-urlencoded payloads.
• The Go client library provides AddDocumentJSON and AddDocumentsJSON methods that handle batching and error checking automatically.
• Successful indexing returns HTTP 201, while skipped documents return HTTP 406.
• Validation occurs in validateAddDocument within server/endpoints.go, ensuring URL integrity before storage.

Frequently Asked Questions

What is the maximum number of documents per batch request?

The /api/batch endpoint enforces a maximum of 100 operations per request. The Go client library automatically partitions larger slices into multiple compliant requests, but raw HTTP callers must implement this splitting logic manually.

What HTTP status codes does the Hister API return when adding documents?

The API returns HTTP 201 Created when a document is successfully indexed and stored. It returns HTTP 406 Not Acceptable when the document passes validation but is skipped due to user-defined indexing rules or filters.

Can I add documents using form-encoded data instead of JSON?

Yes. While JSON is recommended, the /api/add endpoint also accepts application/x-www-form-urlencoded content. Map the url, title, and text fields as form parameters when using this format.

How does the Go client handle bulk document insertion?

The AddDocumentsJSON function in client/document.go accepts a slice of *document.Document structs, automatically chunks them into batches of 100 or fewer, and sends sequential POST requests to /api/batch. It returns a slice of results containing individual status codes and error messages for each document.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}