# How Hister Handles File Deletion and Indexing Updates: A Deep Dive into the Source Code

> Explore how Hister handles file deletion and indexing updates by examining its source code. Learn about DeleteFile and DeleteByQuery for efficient index management.

- Repository: [Adam Tauber/hister](https://github.com/asciimoo/hister)
- Tags: deep-dive
- Published: 2026-09-01

---

**Hister coordinates file deletion through three layers: the `DeleteFile` method removes individual documents from the Bleve search index and vector store embeddings, while `DeleteByQuery` handles bulk deletions with optional history cleanup, ensuring consistent state across the SQLite or Postgres backend.**

When a file is removed from a watched directory in the `asciimoo/hister` repository, the indexer must synchronize the search index, vector embeddings, and history tables to maintain data consistency. Understanding how Hister handles file deletion and indexing updates reveals the careful orchestration between the Bleve indexer, vector store implementations, and database models.

## The DeleteFile Method: Single File Removal

The entry point for file deletion is the `DeleteFile` method in the core indexer implementation. This function acts as a coordinator that ensures both the search index and vector embeddings are purged when a file disappears from the filesystem.

Located in [[`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go)](https://github.com/asciimoo/hister/blob/master/server/indexer/indexer.go) at lines 424-438, this method first constructs a query matching the file's URL, then delegates the actual removal work to `DeleteByQuery`. After the Bleve index entries are removed, it explicitly triggers vector store cleanup to eliminate associated embedding chunks.

### Removing from the Bleve Index

When `DeleteFile` invokes `DeleteByQuery`, it passes a query string that targets the specific file path. The indexer executes this against the Bleve search index, collecting all matching document IDs into a batch operation. This batch processing ensures atomic deletion of all index entries associated with the file.

### Cleaning Up Vector Embeddings

After purging the Bleve entries, `DeleteFile` calls the vector store's `Delete` method to remove the corresponding embedding vectors. This dual-phase approach ensures that no orphaned vectors remain in the SQLite or Postgres backend after the file reference disappears from the search index.

```go
// Example: Delete a specific file from the index
func removeFileFromIndex(idx *indexer.Indexer, filePath string) error {
    // filePath is the absolute path of the deleted file
    if err := idx.DeleteFile(filePath); err != nil {
        log.Error().Err(err).Str("path", filePath).Msg("failed to delete file from index")
        return err
    }
    return nil
}

```

## Bulk Deletion with DeleteByQuery

For operations that remove multiple documents based on criteria—such as deleting all files older than 90 days or matching a specific pattern—Hister provides the `DeleteByQuery` method at line 1609 in [[`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go)](https://github.com/asciimoo/hister/blob/master/server/indexer/indexer.go#L1609).

This method implements the heavy-lifting logic for query-based deletions. It parses the user-supplied query string, executes it against the Bleve index, and iterates through matching documents to build a deletion batch.

### Query Parsing and Batch Processing

`DeleteByQuery` first parses the query string using Bleve's query parser, then executes the search to retrieve all matching document IDs. For each match, it adds the document ID to a Bleve batch deletion. If a caller supplies an `onDelete` callback function, the method also invokes history cleanup for each deleted document.

```go
// Example: Delete documents matching a time-based query
deleted, err := idx.DeleteByQuery("updated:>90d", nil, func(url string, uid uint) {
    // Clean up associated history entries
    if err := model.DeleteHistoryURL(uid, url); err != nil {
        log.Warn().Err(err).Str("url", url).Msg("failed to delete history")
    }
})
if err != nil {
    log.Error().Err(err).Msg("bulk delete operation failed")
}
log.Info().Int("deleted_count", deleted).Msg("documents removed from index")

```

### History Cleanup Callback

The optional callback parameter allows `DeleteByQuery` to maintain referential integrity with the history table. When provided, the function receives the document URL and user ID, enabling the removal of corresponding entries via [`model.DeleteHistoryURL`](https://github.com/asciimoo/hister/blob/master/server/model/history.go) before the vector embeddings are purged.

## Vector Store Implementation Details

The vector store provides a unified interface for embedding management across different database backends. Both SQLite and Postgres implementations adhere to the `VectorStore` interface defined in [[`server/vectorstore/vectorstore.go`](https://github.com/asciimoo/hister/blob/main/server/vectorstore/vectorstore.go)](https://github.com/asciimoo/hister/blob/master/server/vectorstore/vectorstore.go), ensuring consistent deletion semantics regardless of the storage engine.

### SQLite Vector Deletion

The SQLite implementation in [[`server/vectorstore/sqlite.go`](https://github.com/asciimoo/hister/blob/main/server/vectorstore/sqlite.go)](https://github.com/asciimoo/hister/blob/master/server/vectorstore/sqlite.go) at lines 221-227 handles deletion by executing a SQL statement that removes all rows matching the document ID. This operation deletes every embedding chunk associated with the specified file.

```go
// Conceptual implementation of sqliteVectorStore.Delete
func (s *sqliteVectorStore) Delete(docID string) error {
    query := `DELETE FROM vector_embeddings WHERE doc_id = ?`
    _, err := s.db.Exec(query, docID)
    return err
}

```

### Postgres Vector Deletion

Similarly, the Postgres implementation in [[`server/vectorstore/postgres.go`](https://github.com/asciimoo/hister/blob/main/server/vectorstore/postgres.go)](https://github.com/asciimoo/hister/blob/master/server/vectorstore/postgres.go) at lines 108-113 performs the same cleanup using a parameterized `DELETE` statement. Both implementations ensure that when a document is removed from the Bleve index, its corresponding high-dimensional vectors are also eliminated from persistent storage.

```go
// Conceptual implementation of pgVectorStore.Delete
func (p *pgVectorStore) Delete(docID string) error {
    query := `DELETE FROM vector_embeddings WHERE doc_id = $1`
    _, err := p.db.Exec(query, docID)
    return err
}

```

## Summary

Hister maintains index consistency during file deletion through a carefully orchestrated pipeline:

- **`DeleteFile`** (lines 424-438 in [`indexer.go`](https://github.com/asciimoo/hister/blob/main/indexer.go)) initiates single-file removal by querying the Bleve index and triggering vector cleanup
- **`DeleteByQuery`** (line 1609 in [`indexer.go`](https://github.com/asciimoo/hister/blob/main/indexer.go)) handles bulk deletions with batch processing and optional history table cleanup via callbacks
- **Vector store implementations** in [`sqlite.go`](https://github.com/asciimoo/hister/blob/main/sqlite.go) (lines 221-227) and [`postgres.go`](https://github.com/asciimoo/hister/blob/main/postgres.go) (lines 108-113) ensure embedding chunks are purged from persistent storage using document ID-based deletion
- **History integration** through `model.DeleteHistoryURL` maintains referential integrity when documents are removed

## Frequently Asked Questions

### What happens to vector embeddings when a file is deleted in Hister?

When a file is deleted, the `DeleteFile` method calls the vector store's `Delete` function with the document ID. Both the SQLite and Postgres implementations delete all embedding chunks associated with that document ID from their respective tables, ensuring no orphaned vectors remain in the database.

### How does Hister handle bulk deletion of old documents?

Hister uses the `DeleteByQuery` method to process bulk deletions based on search queries. The method parses the query, executes it against the Bleve index to collect matching document IDs, and deletes them in batches. It also supports an optional callback function to clean up history entries for each deleted document.

### Where is the file deletion logic implemented in Hister?

The primary file deletion logic resides in [`server/indexer/indexer.go`](https://github.com/asciimoo/hister/blob/main/server/indexer/indexer.go). The `DeleteFile` method appears at lines 424-438, while the underlying `DeleteByQuery` implementation is located at lines 1609-1650. Vector storage deletion is implemented in [`server/vectorstore/sqlite.go`](https://github.com/asciimoo/hister/blob/main/server/vectorstore/sqlite.go) and [`server/vectorstore/postgres.go`](https://github.com/asciimoo/hister/blob/main/server/vectorstore/postgres.go).

### Does Hister remove history entries when deleting indexed files?

Yes, when the `DeleteByQuery` method is invoked with an `onDelete` callback, it triggers `model.DeleteHistoryURL` for each deleted document. This ensures that history entries associated with removed files are also purged from the database, maintaining referential integrity across the application.