# How CyberStrikeAI Uses a Vector Database for Security Knowledge Retrieval

> Explore how CyberStrikeAI leverages a vector database with hybrid search, combining cosine similarity and BM25 keyword scoring, for efficient security knowledge retrieval.

- Repository: [公明/CyberStrikeAI](https://github.com/Ed1s0nZ/CyberStrikeAI)
- Tags: internals
- Published: 2026-03-09

---

**CyberStrikeAI stores security knowledge as dense embeddings in a SQLite-backed vector database and retrieves relevant context using a hybrid search that combines cosine similarity with BM25 keyword scoring.**

CyberStrikeAI implements a specialized vector database architecture to power its security knowledge retrieval system. The repository at `Ed1s0nZ/CyberStrikeAI` persists 1536-dimensional dense vectors in SQLite, enabling semantic search across markdown documentation while maintaining precise keyword matching through a configurable hybrid scoring algorithm.

## Architecture Overview

### Database Schema and Initialization

When the application initializes, `DB.initKnowledgeTables()` in [`internal/database/database.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/database/database.go) (lines 531‑560) creates the storage layer for the vector database. The schema consists of three core tables:

- `knowledge_base_items` – stores raw markdown file paths, categories, titles, and full content.
- `knowledge_embeddings` – the **vector store** containing chunks and their embeddings as JSON-encoded text (`embedding TEXT NOT NULL`).
- `knowledge_retrieval_logs` – optional audit trail for query tracking.

Vectors are stored as serialized JSON arrays of float32 values alongside the original chunk text, an `item_id` referencing the parent document, and an auto-generated `chunk_index` for ordering.

### Knowledge Ingestion and Indexing

The ingestion pipeline begins with `Manager.ScanKnowledgeBase()` in [`internal/knowledge/manager.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/knowledge/manager.go) (lines 64‑74). This function walks the configured `knowledge.base_path` (default `knowledge/`), extracting the **risk type** from the first subdirectory and the **title** from the filename. New or modified items are queued for processing.

The `Indexer.IndexItem()` function in [`internal/knowledge/indexer.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/knowledge/indexer.go) (lines 94‑119) handles the actual vectorization:

1. **Chunking** – Splits document content into manageable segments.
2. **Metadata Augmentation** – Prefixes each chunk with category and title context:
   ```go
   textForEmbedding := fmt.Sprintf("[风险类型：%s] [标题：%s]\n%s", category, title, chunk)
   ```

3. **Embedding Generation** – Calls `Embedder.EmbedText()` which wraps the OpenAI `/embeddings` endpoint (see [`internal/knowledge/embedder.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/knowledge/embedder.go), lines 64‑84).
4. **Persistence** – Marshals the `[]float32` vector to JSON and inserts it into the `knowledge_embeddings` table (lines 74‑82).

### Hybrid Retrieval Mechanism

Retrieval is implemented in [`internal/knowledge/retriever.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/knowledge/retriever.go) using a dual-track scoring system. When a user submits a query, the system executes two parallel search strategies:

1. **Vector Similarity** – Computes **cosine similarity** between the query embedding and all stored vectors (implementation in lines 52‑60). The system streams vectors from the database, filtering by category if a risk type is specified.
2. **Keyword Matching** – Calculates **BM25 scores** across the raw chunk text, category, and title fields.

These scores are fused using a configurable hybrid weight:

```go
hybridScore = config.SimilarityWeight*vectorScore + (1-config.SimilarityWeight)*bm25Score

```

The `SimilarityWeight` value defaults to **0.7** (70% vector, 30% keyword) and is read from [`config.yaml`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/config.yaml) (line 118). Additional filtering logic in lines 150‑240 of [`retriever.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/retriever.go) applies dynamic similarity thresholds and cross-language relaxations to ensure relevant results surface even with terminology variations.

## Implementation Details

### Embedding Generation with Metadata

The `Embedder` struct in [`internal/knowledge/embedder.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/knowledge/embedder.go) manages communication with the OpenAI API, implementing rate limiting and retry logic. The critical method `doEmbedText()` (lines 64‑84) handles the actual API request, returning 1536-dimensional vectors suitable for the SQLite storage format.

By prefixing chunks with risk type and title metadata before embedding (as shown in the `textForEmbedding` format string), CyberStrikeAI ensures that semantic search respects document categorization without requiring separate metadata filters in the vector space.

### Similarity Calculation and Threshold Logic

The cosine similarity implementation compares query vectors against the stored JSON vectors:

```go
// Conceptual implementation from retriever.go lines 52-60
func cosineSimilarity(a, b []float32) float64 {
    // Dot product divided by magnitudes
    // Returns 0.0 to 1.0 where 1.0 is identical
}

```

Results are filtered against dynamic thresholds, with fallback "top‑K" logic ensuring the system returns the most relevant chunks even when strict similarity cutoffs eliminate all candidates.

## Configuration and Usage Examples

### Adjusting Hybrid Search Weights

Tuning the balance between semantic and lexical search requires no code changes. Modify [`config.yaml`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/config.yaml):

```yaml
knowledge:
  hybrid_weight: 0.5   # Equal weight to vector and BM25 scores

```

Values range from `0.0` (pure BM25 keyword search) to `1.0` (pure vector similarity), with the default `0.7` favoring semantic matching.

### Programmatic Knowledge Indexing

To index new security documentation:

```go
// Initialize manager with database connection
mgr := knowledge.NewManager(db, "./knowledge", logger)

// Detect new or changed files
toIndex, err := mgr.ScanKnowledgeBase()
if err != nil { log.Fatal(err) }

// Process embeddings
idx := knowledge.NewIndexer(db, embedder, logger, cfg.Knowledge.Indexing)
for _, itemID := range toIndex {
    if err := idx.IndexItem(context.Background(), itemID); err != nil {
        logger.Error("index failed", zap.String("item", itemID), zap.Error(err))
    }
}

```

This workflow automatically extracts categories from folder structure and titles from filenames, then persists dense vectors to the SQLite store.

### Executing Hybrid Queries

Querying the knowledge base combines the embedding client and retrieval engine:

```go
// Configure retriever
retr := knowledge.NewRetriever(db, embedder, logger, cfg.Knowledge.Retrieval)

// Build request with optional risk type filtering
req := knowledge.RetrievalRequest{
    Query:    "What are mitigation steps for SQL injection?",
    RiskType: "sql-injection", // Matches category folder name
    TopK:     5,
}

// Execute hybrid search
result, err := retr.Retrieve(context.Background(), req)
if err != nil { log.Fatal(err) }

for _, r := range result.Items {
    fmt.Printf("- %s (score: %.2f)\n", r.Chunk.ChunkText, r.Score)
}

```

The `Retrieve` method automatically embeds the query, computes cosine similarity against all stored vectors, calculates BM25 scores, applies the hybrid weighting, and returns ranked results via the web handler in [`internal/handler/knowledge.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/handler/knowledge.go).

## Summary

- **SQLite-backed storage** persists 1536-dimensional vectors as JSON text in the `knowledge_embeddings` table, eliminating external database dependencies.
- **Metadata-augmented embeddings** incorporate risk categories and document titles directly into the vector space, improving retrieval accuracy for security-specific queries.
- **Hybrid scoring algorithm** combines cosine similarity with BM25 keyword matching using a configurable weight (default 0.7) to balance semantic understanding with precise term matching.
- **Automatic change detection** via `Manager.ScanKnowledgeBase()` enables incremental indexing of markdown files without full re-ingestion.
- **Provider-agnostic embedder** supports custom embedding endpoints beyond OpenAI through configuration in [`config.yaml`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/config.yaml).

## Frequently Asked Questions

### What database engine does CyberStrikeAI use for vector storage?

CyberStrikeAI uses **SQLite** as its vector database engine. Rather than deploying a specialized vector database like Pinecone or Weaviate, the system stores 1536-dimensional embeddings as JSON-encoded text in the `knowledge_embeddings` table. This design simplifies deployment while maintaining efficient retrieval through Go-based similarity calculations.

### How does CyberStrikeAI incorporate document metadata into vector search?

During the indexing phase in [`internal/knowledge/indexer.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/knowledge/indexer.go), each content chunk is prefixed with structured metadata before embedding: `[风险类型：{category}] [标题：{title}]\n{chunk}`. This ensures that the risk type and document title participate in the semantic vector space, allowing queries like "mitigation for SQL injection" to match documents in the `sql-injection` category even if the exact phrase doesn't appear in the chunk text.

### Can I adjust the balance between semantic similarity and keyword matching?

Yes. The hybrid weight parameter in [`config.yaml`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/config.yaml) (line 118) controls the fusion of vector similarity and BM25 scores. Setting `hybrid_weight: 1.0` uses pure cosine similarity, while `0.0` relies solely on BM25 keyword matching. The default value of `0.7` provides strong semantic search capabilities while preserving keyword precision for specific security terminology.

### What happens if no results meet the similarity threshold?

The retriever in [`internal/knowledge/retriever.go`](https://github.com/Ed1s0nZ/CyberStrikeAI/blob/main/internal/knowledge/retriever.go) (lines 150‑240) implements dynamic threshold logic with fallback mechanisms. If no candidates pass the strict similarity cutoff, the system applies cross-language relaxations and ultimately falls back to a "top‑K" strategy, ensuring the LLM receives relevant context rather than empty results. This logic is exposed through the `/api/knowledge/query` endpoint used by the web interface.