How Hister Re-Indexes Documents When the Embedding Configuration Changes

Hister detects embedding configuration changes by comparing a SHA-256 fingerprint of the current SemanticSearch settings against a stored fingerprint in the index metadata, automatically triggering a full re-index via ReindexAll() when mismatches occur.

When you modify embedding parameters such as the model, dimensions, or chunk overlap in asciimoo/hister, the existing vector embeddings become incompatible with your new configuration. Hister handles this automatically by computing a unique fingerprint of the embedding settings and comparing it against the index metadata on startup, initiating a full re-index when discrepancies are detected.

How Hister Detects Embedding Configuration Changes

Computing the Embedding Fingerprint

In config/config.go, the SemanticSearch.EmbeddingFingerprint() method generates a deterministic SHA-256 hash based on all embedding-related configuration fields. This uniquely identifies the embedding configuration by hashing the endpoint, model name, dimensions, maximum context length, chunk overlap, and document prefix.

// In config/config.go
func (s SemanticSearch) EmbeddingFingerprint() string {
    if !s.Enable { return "" }
    payload := struct {
        Version          int    `json:"version"`
        Endpoint         string `json:"endpoint"`
        Model            string `json:"model"`
        Dimensions       int    `json:"dimensions"`
        MaxContextLength int    `json:"max_context_length"`
        ChunkOverlap     int    `json:"chunk_overlap"`
        DocumentPrefix   string `json:"document_prefix"`
    }{
        Version:          1,
        Endpoint:         s.EmbeddingEndpoint,
        Model:            s.EmbeddingModel,
        Dimensions:       s.Dimensions,
        MaxContextLength: s.MaxContextLength,
        ChunkOverlap:     s.ChunkOverlap,
        DocumentPrefix:   s.DocumentPrefix,
    }
    data, _ := json.Marshal(payload)
    sum := sha256.Sum256(data)
    return hex.EncodeToString(sum[:])
}

Storing Fingerprint Metadata

When an index is created, the fingerprint is persisted to the index metadata via backfillEmbeddingFingerprint in server/indexer/metadata.go (lines 107-119). The GetEmbeddingFingerprint function retrieves this stored value on subsequent startups (lines 100-105).

The Automatic Re-Indexing Workflow

Detecting Configuration Drift at Startup

The root command in cmd/root.go orchestrates the comparison between the stored fingerprint and the active configuration. During initialization, it loads the stored value using idx.GetEmbeddingFingerprint() and computes the current fingerprint via cfg.SemanticSearch.EmbeddingFingerprint(). If the strings differ, Hister generates a warning and invokes idx.ReindexAll().

// In cmd/root.go
storedEmbeddingFingerprint, _ := idx.GetEmbeddingFingerprint()
activeEmbeddingFingerprint := cfg.SemanticSearch.EmbeddingFingerprint()

if warning := embeddingConfigWarning(storedEmbeddingFingerprint,
                                    activeEmbeddingFingerprint); warning != "" {
    fmt.Fprintln(os.Stderr, warning)
    // Force a full re-index
    if err := idx.ReindexAll(context.Background()); err != nil {
        log.Fatal(err)
    }
}

Executing the Full Re-Index

The ReindexAll function in server/indexer/update.go iterates through every document in the index, enqueuing embedding jobs for each document ID. The embedding queue workers, started via startEmbeddingQueue, process these jobs by calling the external embedder and writing new vectors to the vector store.

// In server/indexer/update.go
func (i *Indexer) ReindexAll(ctx context.Context) error {
    // Walk every stored document
    for _, doc := range i.allDocuments() {
        // Force embedding even if it was already present
        if err := i.enqueueEmbedding(doc.ID()); err != nil {
            return err
        }
    }
    // Wait for the embedding workers to finish
    i.waitEmbeddingQueue()
    return nil
}

The startEmbeddingQueue function initializes worker goroutines that pull from the embedding channel and process each document:

// In server/indexer/update.go – startEmbeddingQueue
func (i *Indexer) startEmbeddingQueue(workers int) error {
    i.embedQueue = make(chan string, workers)
    for n := 0; n < workers; n++ {
        go func() {
            for id := range i.embedQueue {
                // Load document, generate embedding, store it
                _ = i.processEmbedding(id)
            }
        }()
    }
    return nil
}

Persisting the New Configuration

After the re-index completes successfully, Hister updates the stored fingerprint in the metadata using setMetadata in server/indexer/metadata.go. This ensures subsequent startups recognize the configuration as current and do not trigger redundant re-indexing.

Key Implementation Details

• config/config.go: Defines the SemanticSearch struct and implements EmbeddingFingerprint() to generate configuration hashes.
• server/indexer/metadata.go: Handles reading and writing the fingerprint via GetEmbeddingFingerprint, backfillEmbeddingFingerprint, and setMetadata.
• cmd/root.go (lines 549-587): Contains the comparison logic that triggers re-indexing when fingerprints mismatch.
• server/indexer/update.go: Implements the ReindexAll logic, embedding queue management, and worker pool orchestration.

Summary

• Hister computes a SHA-256 fingerprint of embedding configuration fields in EmbeddingFingerprint() within config/config.go.
• The fingerprint is stored in index metadata and validated on every application startup.
• Mismatches between stored and active fingerprints automatically trigger ReindexAll() without manual intervention.
• The re-index process regenerates embeddings for all documents using worker pools and queue-based processing.
• After completion, the new fingerprint is persisted to metadata to prevent duplicate re-indexing cycles.

Frequently Asked Questions

What specific embedding settings trigger a re-index in Hister?

Any change to the SemanticSearch configuration fields included in the fingerprint calculation triggers a re-index. According to the source code in config/config.go, these fields include the embedding endpoint, model name, dimensions, max context length, chunk overlap, and document prefix.

Where does Hister store the embedding configuration fingerprint?

The fingerprint is stored in the index metadata file managed by server/indexer/metadata.go. The GetEmbeddingFingerprint function retrieves the stored value, while backfillEmbeddingFingerprint and setMetadata handle writing the fingerprint during index creation and after re-indexing completes.

How does Hister perform the actual re-indexing of documents?

Hister calls ReindexAll() defined in server/indexer/update.go, which iterates through all documents and enqueues them for re-embedding via enqueueEmbedding(). Worker goroutines started by startEmbeddingQueue process these jobs asynchronously, generating new vectors using the updated configuration and storing them in the vector database.

Is the re-indexing process automatic or does it require manual intervention?

The process is fully automatic. When cmd/root.go detects a fingerprint mismatch during application startup, it prints a warning via embeddingConfigWarning and immediately initiates the re-index by calling idx.ReindexAll() without requiring user action or confirmation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}