Even as modern LLM context windows reach millions of tokens, piping an entire enterprise codebase into every prompt is economically ruinous and induces profound attention degradation. Kernax's hybrid indexing engine synthesizes BM25 lexical search, Tree-Sitter symbol graphs, and dense vector embeddings via Reciprocal Rank Fusion (RRF) to retrieve the precise 500 lines required in under 15 milliseconds.
MAX_BUFFER_REALLOC_BYTES); pure keyword matching fails on abstract intent (e.g., "handle token expiration gracefully"). Kernax reconciles both with AST dependency paths to achieve 98.4% Recall@10.
Why Naive RAG Fails in Source Code Repositories
Standard document-based RAG implementations divide files into arbitrary 500-character segments. In source code, this creates catastrophic architectural blind spots:
- Syntactic Fragmentation: A 40-line function gets bifurcated across chunk boundaries, separating method signatures from return statements and error catches.
- Distant Dependency Blindness: A critical domain type may reside in
packages/core/types/session.tswhile the implementation is inservices/auth/handler.go. Flat vector chunks fail to bridge this explicit topological relationship. - Mock Contamination: When a symbol appears across dozens of unit tests and fixtures, naive search retrieves mock implementations rather than the production business logic.
The Three-Tier Kernax Hybrid Index
Running continuously in an asynchronous, low-priority background thread with minimal CPU utilization, the Kernax indexer unifies three distinct primitives:
- AST Symbol Dependency Graph: Every file is parsed into Tree-Sitter concrete syntax trees to catalog classes, methods, imports, and call sites. When an agent queries a handler, the underlying interface definitions are hoisted automatically.
- Lexical Inverted Index (BM25): An embedded SQLite FTS5 index enables microsecond exact-token queries across identifiers, configuration keys, and error constants.
- Dense Vector Embeddings: A compact local embedding model encodes semantic meaning without transmitting raw source files to external endpoints.
Reciprocal Rank Fusion (RRF) Mathematical Formulation
Results from disparate candidate generators are harmonized using weighted Reciprocal Rank Fusion:
// RRF Fusion Formula
RRF_Score(d) = Σ ( w_m / (k + rank_m(d)) )
Where:
- m ∈ {BM25_Lexical, Dense_Vector, AST_Graph}
- k = 60 (Smoothing factor dampening low-rank outliers)
- w_AST = 1.4 (Heightened weight for concrete call-graph relationships)
- w_BM25 = 1.1 (Prioritizing exact identifier hits)
- w_Vector = 0.9 (Conceptual intent weighting)Empirical Benchmark on 2.5M LOC Repository
Performance evaluated against a monorepo consisting of 2.5 million lines of TypeScript, Go, and SQL:
| Indexing Approach | Retrieval Accuracy (Recall@10) | Query Latency (P95) | Local RAM Overhead |
|---|---|---|---|
| Pure Vector Search (Cloud Embeddings) | 64.2% | 380ms | Cloud API Overhead |
| Standard BM25 / Ripgrep | 71.8% | 45ms | 40 MB |
| Kernax Hybrid RRF + AST Graph | 98.4% | 14ms | 85 MB (Local SQLite) |
Absolute Data Privacy: Local-First Index Storage
Crucially, the entire index resides locally on your workstation at .kernax/index.db. Your intellectual property is never dumped into third-party vector databases or stored on remote servers. Only the tightly filtered, relevant code slices are sent to the reasoning model upon explicit user prompt.