How Hister Handles File Deletion and Indexing Updates: A Deep Dive into the Source Code

Hister coordinates file deletion through three layers: the DeleteFile method removes individual documents from the Bleve search index and vector store embeddings, while DeleteByQuery handles bulk deletions with optional history cleanup, ensuring consistent state across the SQLite or Postgres backend.

When a file is removed from a watched directory in the asciimoo/hister repository, the indexer must synchronize the search index, vector embeddings, and history tables to maintain data consistency. Understanding how Hister handles file deletion and indexing updates reveals the careful orchestration between the Bleve indexer, vector store implementations, and database models.

The DeleteFile Method: Single File Removal

The entry point for file deletion is the DeleteFile method in the core indexer implementation. This function acts as a coordinator that ensures both the search index and vector embeddings are purged when a file disappears from the filesystem.

Located in [server/indexer/indexer.go](https://github.com/asciimoo/hister/blob/master/server/indexer/indexer.go) at lines 424-438, this method first constructs a query matching the file's URL, then delegates the actual removal work to DeleteByQuery. After the Bleve index entries are removed, it explicitly triggers vector store cleanup to eliminate associated embedding chunks.

Removing from the Bleve Index

When DeleteFile invokes DeleteByQuery, it passes a query string that targets the specific file path. The indexer executes this against the Bleve search index, collecting all matching document IDs into a batch operation. This batch processing ensures atomic deletion of all index entries associated with the file.

Cleaning Up Vector Embeddings

After purging the Bleve entries, DeleteFile calls the vector store's Delete method to remove the corresponding embedding vectors. This dual-phase approach ensures that no orphaned vectors remain in the SQLite or Postgres backend after the file reference disappears from the search index.

// Example: Delete a specific file from the index
func removeFileFromIndex(idx *indexer.Indexer, filePath string) error {
    // filePath is the absolute path of the deleted file
    if err := idx.DeleteFile(filePath); err != nil {
        log.Error().Err(err).Str("path", filePath).Msg("failed to delete file from index")
        return err
    }
    return nil
}

Bulk Deletion with DeleteByQuery

For operations that remove multiple documents based on criteria—such as deleting all files older than 90 days or matching a specific pattern—Hister provides the DeleteByQuery method at line 1609 in [server/indexer/indexer.go](https://github.com/asciimoo/hister/blob/master/server/indexer/indexer.go#L1609).

This method implements the heavy-lifting logic for query-based deletions. It parses the user-supplied query string, executes it against the Bleve index, and iterates through matching documents to build a deletion batch.

Query Parsing and Batch Processing

DeleteByQuery first parses the query string using Bleve's query parser, then executes the search to retrieve all matching document IDs. For each match, it adds the document ID to a Bleve batch deletion. If a caller supplies an onDelete callback function, the method also invokes history cleanup for each deleted document.

// Example: Delete documents matching a time-based query
deleted, err := idx.DeleteByQuery("updated:>90d", nil, func(url string, uid uint) {
    // Clean up associated history entries
    if err := model.DeleteHistoryURL(uid, url); err != nil {
        log.Warn().Err(err).Str("url", url).Msg("failed to delete history")
    }
})
if err != nil {
    log.Error().Err(err).Msg("bulk delete operation failed")
}
log.Info().Int("deleted_count", deleted).Msg("documents removed from index")

History Cleanup Callback

The optional callback parameter allows DeleteByQuery to maintain referential integrity with the history table. When provided, the function receives the document URL and user ID, enabling the removal of corresponding entries via model.DeleteHistoryURL before the vector embeddings are purged.

Vector Store Implementation Details

The vector store provides a unified interface for embedding management across different database backends. Both SQLite and Postgres implementations adhere to the VectorStore interface defined in [server/vectorstore/vectorstore.go](https://github.com/asciimoo/hister/blob/master/server/vectorstore/vectorstore.go), ensuring consistent deletion semantics regardless of the storage engine.

SQLite Vector Deletion

The SQLite implementation in [server/vectorstore/sqlite.go](https://github.com/asciimoo/hister/blob/master/server/vectorstore/sqlite.go) at lines 221-227 handles deletion by executing a SQL statement that removes all rows matching the document ID. This operation deletes every embedding chunk associated with the specified file.

// Conceptual implementation of sqliteVectorStore.Delete
func (s *sqliteVectorStore) Delete(docID string) error {
    query := `DELETE FROM vector_embeddings WHERE doc_id = ?`
    _, err := s.db.Exec(query, docID)
    return err
}

Postgres Vector Deletion

Similarly, the Postgres implementation in [server/vectorstore/postgres.go](https://github.com/asciimoo/hister/blob/master/server/vectorstore/postgres.go) at lines 108-113 performs the same cleanup using a parameterized DELETE statement. Both implementations ensure that when a document is removed from the Bleve index, its corresponding high-dimensional vectors are also eliminated from persistent storage.

// Conceptual implementation of pgVectorStore.Delete
func (p *pgVectorStore) Delete(docID string) error {
    query := `DELETE FROM vector_embeddings WHERE doc_id = $1`
    _, err := p.db.Exec(query, docID)
    return err
}

Summary

Hister maintains index consistency during file deletion through a carefully orchestrated pipeline:

• DeleteFile (lines 424-438 in indexer.go) initiates single-file removal by querying the Bleve index and triggering vector cleanup
• DeleteByQuery (line 1609 in indexer.go) handles bulk deletions with batch processing and optional history table cleanup via callbacks
• Vector store implementations in sqlite.go (lines 221-227) and postgres.go (lines 108-113) ensure embedding chunks are purged from persistent storage using document ID-based deletion
• History integration through model.DeleteHistoryURL maintains referential integrity when documents are removed

Frequently Asked Questions

What happens to vector embeddings when a file is deleted in Hister?

When a file is deleted, the DeleteFile method calls the vector store's Delete function with the document ID. Both the SQLite and Postgres implementations delete all embedding chunks associated with that document ID from their respective tables, ensuring no orphaned vectors remain in the database.

How does Hister handle bulk deletion of old documents?

Hister uses the DeleteByQuery method to process bulk deletions based on search queries. The method parses the query, executes it against the Bleve index to collect matching document IDs, and deletes them in batches. It also supports an optional callback function to clean up history entries for each deleted document.

Where is the file deletion logic implemented in Hister?

The primary file deletion logic resides in server/indexer/indexer.go. The DeleteFile method appears at lines 424-438, while the underlying DeleteByQuery implementation is located at lines 1609-1650. Vector storage deletion is implemented in server/vectorstore/sqlite.go and server/vectorstore/postgres.go.

Does Hister remove history entries when deleting indexed files?

Yes, when the DeleteByQuery method is invoked with an onDelete callback, it triggers model.DeleteHistoryURL for each deleted document. This ensures that history entries associated with removed files are also purged from the database, maintaining referential integrity across the application.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}