How CyberStrikeAI Uses a Vector Database for Security Knowledge Retrieval

CyberStrikeAI stores security knowledge as dense embeddings in a SQLite-backed vector database and retrieves relevant context using a hybrid search that combines cosine similarity with BM25 keyword scoring.

CyberStrikeAI implements a specialized vector database architecture to power its security knowledge retrieval system. The repository at Ed1s0nZ/CyberStrikeAI persists 1536-dimensional dense vectors in SQLite, enabling semantic search across markdown documentation while maintaining precise keyword matching through a configurable hybrid scoring algorithm.

Architecture Overview

Database Schema and Initialization

When the application initializes, DB.initKnowledgeTables() in internal/database/database.go (lines 531‑560) creates the storage layer for the vector database. The schema consists of three core tables:

• knowledge_base_items – stores raw markdown file paths, categories, titles, and full content.
• knowledge_embeddings – the vector store containing chunks and their embeddings as JSON-encoded text (embedding TEXT NOT NULL).
• knowledge_retrieval_logs – optional audit trail for query tracking.

Vectors are stored as serialized JSON arrays of float32 values alongside the original chunk text, an item_id referencing the parent document, and an auto-generated chunk_index for ordering.

Knowledge Ingestion and Indexing

The ingestion pipeline begins with Manager.ScanKnowledgeBase() in internal/knowledge/manager.go (lines 64‑74). This function walks the configured knowledge.base_path (default knowledge/), extracting the risk type from the first subdirectory and the title from the filename. New or modified items are queued for processing.

The Indexer.IndexItem() function in internal/knowledge/indexer.go (lines 94‑119) handles the actual vectorization:

1. Chunking – Splits document content into manageable segments.

2. Metadata Augmentation – Prefixes each chunk with category and title context:

   textForEmbedding := fmt.Sprintf("[风险类型：%s] [标题：%s]\n%s", category, title, chunk)

3. Embedding Generation – Calls Embedder.EmbedText() which wraps the OpenAI /embeddings endpoint (see internal/knowledge/embedder.go, lines 64‑84).

4. Persistence – Marshals the []float32 vector to JSON and inserts it into the knowledge_embeddings table (lines 74‑82).

Hybrid Retrieval Mechanism

Retrieval is implemented in internal/knowledge/retriever.go using a dual-track scoring system. When a user submits a query, the system executes two parallel search strategies:

1. Vector Similarity – Computes cosine similarity between the query embedding and all stored vectors (implementation in lines 52‑60). The system streams vectors from the database, filtering by category if a risk type is specified.
2. Keyword Matching – Calculates BM25 scores across the raw chunk text, category, and title fields.

These scores are fused using a configurable hybrid weight:

hybridScore = config.SimilarityWeight*vectorScore + (1-config.SimilarityWeight)*bm25Score

The SimilarityWeight value defaults to 0.7 (70% vector, 30% keyword) and is read from config.yaml (line 118). Additional filtering logic in lines 150‑240 of retriever.go applies dynamic similarity thresholds and cross-language relaxations to ensure relevant results surface even with terminology variations.

Implementation Details

Embedding Generation with Metadata

The Embedder struct in internal/knowledge/embedder.go manages communication with the OpenAI API, implementing rate limiting and retry logic. The critical method doEmbedText() (lines 64‑84) handles the actual API request, returning 1536-dimensional vectors suitable for the SQLite storage format.

By prefixing chunks with risk type and title metadata before embedding (as shown in the textForEmbedding format string), CyberStrikeAI ensures that semantic search respects document categorization without requiring separate metadata filters in the vector space.

Similarity Calculation and Threshold Logic

The cosine similarity implementation compares query vectors against the stored JSON vectors:

// Conceptual implementation from retriever.go lines 52-60
func cosineSimilarity(a, b []float32) float64 {
    // Dot product divided by magnitudes
    // Returns 0.0 to 1.0 where 1.0 is identical
}

Results are filtered against dynamic thresholds, with fallback "top‑K" logic ensuring the system returns the most relevant chunks even when strict similarity cutoffs eliminate all candidates.

Configuration and Usage Examples

Adjusting Hybrid Search Weights

Tuning the balance between semantic and lexical search requires no code changes. Modify config.yaml:

knowledge:
  hybrid_weight: 0.5   # Equal weight to vector and BM25 scores

Values range from 0.0 (pure BM25 keyword search) to 1.0 (pure vector similarity), with the default 0.7 favoring semantic matching.

Programmatic Knowledge Indexing

To index new security documentation:

// Initialize manager with database connection
mgr := knowledge.NewManager(db, "./knowledge", logger)

// Detect new or changed files
toIndex, err := mgr.ScanKnowledgeBase()
if err != nil { log.Fatal(err) }

// Process embeddings
idx := knowledge.NewIndexer(db, embedder, logger, cfg.Knowledge.Indexing)
for _, itemID := range toIndex {
    if err := idx.IndexItem(context.Background(), itemID); err != nil {
        logger.Error("index failed", zap.String("item", itemID), zap.Error(err))
    }
}

This workflow automatically extracts categories from folder structure and titles from filenames, then persists dense vectors to the SQLite store.

Executing Hybrid Queries

Querying the knowledge base combines the embedding client and retrieval engine:

// Configure retriever
retr := knowledge.NewRetriever(db, embedder, logger, cfg.Knowledge.Retrieval)

// Build request with optional risk type filtering
req := knowledge.RetrievalRequest{
    Query:    "What are mitigation steps for SQL injection?",
    RiskType: "sql-injection", // Matches category folder name
    TopK:     5,
}

// Execute hybrid search
result, err := retr.Retrieve(context.Background(), req)
if err != nil { log.Fatal(err) }

for _, r := range result.Items {
    fmt.Printf("- %s (score: %.2f)\n", r.Chunk.ChunkText, r.Score)
}

The Retrieve method automatically embeds the query, computes cosine similarity against all stored vectors, calculates BM25 scores, applies the hybrid weighting, and returns ranked results via the web handler in internal/handler/knowledge.go.

Summary

• SQLite-backed storage persists 1536-dimensional vectors as JSON text in the knowledge_embeddings table, eliminating external database dependencies.
• Metadata-augmented embeddings incorporate risk categories and document titles directly into the vector space, improving retrieval accuracy for security-specific queries.
• Hybrid scoring algorithm combines cosine similarity with BM25 keyword matching using a configurable weight (default 0.7) to balance semantic understanding with precise term matching.
• Automatic change detection via Manager.ScanKnowledgeBase() enables incremental indexing of markdown files without full re-ingestion.
• Provider-agnostic embedder supports custom embedding endpoints beyond OpenAI through configuration in config.yaml.

Frequently Asked Questions

What database engine does CyberStrikeAI use for vector storage?

CyberStrikeAI uses SQLite as its vector database engine. Rather than deploying a specialized vector database like Pinecone or Weaviate, the system stores 1536-dimensional embeddings as JSON-encoded text in the knowledge_embeddings table. This design simplifies deployment while maintaining efficient retrieval through Go-based similarity calculations.

How does CyberStrikeAI incorporate document metadata into vector search?

During the indexing phase in internal/knowledge/indexer.go, each content chunk is prefixed with structured metadata before embedding: [风险类型：{category}] [标题：{title}]\n{chunk}. This ensures that the risk type and document title participate in the semantic vector space, allowing queries like "mitigation for SQL injection" to match documents in the sql-injection category even if the exact phrase doesn't appear in the chunk text.

Can I adjust the balance between semantic similarity and keyword matching?

Yes. The hybrid weight parameter in config.yaml (line 118) controls the fusion of vector similarity and BM25 scores. Setting hybrid_weight: 1.0 uses pure cosine similarity, while 0.0 relies solely on BM25 keyword matching. The default value of 0.7 provides strong semantic search capabilities while preserving keyword precision for specific security terminology.

What happens if no results meet the similarity threshold?

The retriever in internal/knowledge/retriever.go (lines 150‑240) implements dynamic threshold logic with fallback mechanisms. If no candidates pass the strict similarity cutoff, the system applies cross-language relaxations and ultimately falls back to a "top‑K" strategy, ensuring the LLM receives relevant context rather than empty results. This logic is exposed through the /api/knowledge/query endpoint used by the web interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}