vector-db-rag-expert ยท diff
git:20260816.d23736e to git:20260907.82c61cd
168 added, 30 removed. Audit A to A.
---
name: vector-db-rag-expert
- description: "Expert guide for high-performance Vector Databases, RAG architectures, pgvector HNSW indexing, hybrid search (Dense + BM25), and semantic chunking / Panduan ahli Vector DB, arsitektur RAG, pgvector HNSW, dan hybrid search."
- author: "vibes-plug-swarm"
+ description: "Expert guide for high-performance Vector Databases, Deep RAG architectures, pgvector 0.8+ HNSW, Reciprocal Rank Fusion (RRF), Cross-Encoder Re-ranking, and Late Chunking / Panduan ahli Vector DB, arsitektur Deep RAG, pgvector HNSW, RRF, dan Re-ranking."
+ author: "Roedy Rustam"
---
- # Vector DB & Deep RAG Expert
+ # Vector DB & Deep RAG Expert (2026 Edition)
[English](#english) | [Bahasa Indonesia](#bahasa-indonesia)
---
<a name="english"></a>
## English
### Purpose & Overview
- Production-grade guidelines for Vector Databases (pgvector, Qdrant, Pinecone, Milvus), RAG indexing strategies, HNSW vector search, hybrid retrieval (dense vector embeddings + BM25 sparse keyword ranking), semantic document chunking, and RAG evaluation frameworks.
+ Production-grade architectural guide for Vector Databases (PostgreSQL `pgvector 0.8+`, Qdrant, LanceDB, Pinecone), Deep RAG indexing strategies, HNSW iterative search, **Reciprocal Rank Fusion (RRF)** hybrid retrieval, **Cross-Encoder Re-ranking** (Cohere Rerank v3, FlashRank, BGE-Reranker-v2), and **Late Chunking** to eliminate context fragmentation.
### Key Capabilities
- - **pgvector & Hybrid Search**: PostgreSQL `pgvector` HNSW indexing, cosine/L2 distance metric tuning, and BM25 hybrid re-ranking.
- - **RAG Architecture**: Parent-Document retrieval, Hypothetical Document Embeddings (HyDE), and contextual compression.
- - **RAG Evaluation**: Automated retrieval quality scoring using Ragas and TruLens.
+ 1. **pgvector 0.8+ & HNSW Indexing**: High-dimensional vector storage, cosine/inner-product/L2 distance metric tuning, and iterative HNSW index scans with metadata filtering.
+ 2. **Reciprocal Rank Fusion (RRF)**: Combining sparse keyword BM25 ranks with dense semantic vector ranks using $RRF(d) = \sum \frac{1}{k + rank(d)}$, far outperforming naive linear score weighting.
+ 3. **Cross-Encoder Re-ranking**: Two-stage retrieval pipeline: retrieve Top-50 candidates via fast hybrid search, then re-rank down to Top-5 using a cross-encoder model to maximize NDCG@10.
+ 4. **Late Chunking & Contextual Retrieval**: Embed long-context documents in full before pooling token embeddings into individual chunks, preserving document-level semantics across boundaries.
+ 5. **RAG Evaluation**: Continuous retrieval precision and hallucination scoring using automated eval harnesses (Ragas, TruLens, DeepEval).
+ ---
+
+ ### Production Implementation Recipes
+
+ #### Recipe 1: Reciprocal Rank Fusion (RRF) Hybrid Search with Drizzle ORM
```typescript
import { sql } from 'drizzle-orm';
+ import { db } from '@/lib/db';
- // Hybrid Search: Vector Cosine Similarity + Full Text Search
- export async function hybridSearch(queryVector: number[], queryText: string, limit = 10) {
- const result = await db.execute(sql`
- SELECT id, title, content,
- (1 - (embedding <=> ${JSON.stringify(queryVector)}::vector)) * 0.7 +
- ts_rank(fts, websearch_to_tsquery('english', ${queryText})) * 0.3 AS score
- FROM documents
- ORDER BY score DESC
+ export interface SearchResult {
+ id: string;
+ content: string;
+ score: number;
+ }
+
+ /**
+ * Executes Reciprocal Rank Fusion (RRF) combining BM25 keyword search and pgvector HNSW
+ * k = 60 is the industry standard constant
+ */
+ export async function reciprocalRankFusionSearch(
+ queryVector: number[],
+ queryText: string,
+ limit = 10,
+ k = 60
+ ): Promise<SearchResult[]> {
+ const formattedVector = JSON.stringify(queryVector);
+
+ const results = await db.execute(sql`
+ WITH vector_matches AS (
+ SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> ${formattedVector}::vector) AS rank
+ FROM documents
+ WHERE status = 'published'
+ ORDER BY embedding <=> ${formattedVector}::vector
+ LIMIT 50
+ ),
+ text_matches AS (
+ SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', ${queryText})) DESC) AS rank
+ FROM documents
+ WHERE fts @@ websearch_to_tsquery('english', ${queryText})
+ LIMIT 50
+ )
+ SELECT
+ d.id,
+ d.content,
+ COALESCE(1.0 / (${k} + v.rank), 0.0) +
+ COALESCE(1.0 / (${k} + t.rank), 0.0) AS rrf_score
+ FROM documents d
+ LEFT JOIN vector_matches v ON d.id = v.id
+ LEFT JOIN text_matches t ON d.id = t.id
+ WHERE v.id IS NOT NULL OR t.id IS NOT NULL
+ ORDER BY rrf_score DESC
LIMIT ${limit};
`);
- return result;
+
+ return results.rows as unknown as SearchResult[];
}
```
+ #### Recipe 2: Two-Stage Re-Ranking Pipeline with FlashRank (Node.js / TypeScript)
+ ```typescript
+ import { FlashRankRegistry } from 'flashrank';
+
+ const ranker = new FlashRankRegistry();
+
+ export async function rerankCandidates(query: string, candidates: { id: string; text: string }[], topN = 5) {
+ const passages = candidates.map(c => ({ id: c.id, text: c.text }));
+
+ // Ultra-fast client/server cross-encoder re-ranking
+ const reranked = await ranker.rerank({
+ query,
+ passages,
+ model: 'ms-marco-TinyBERT-L-2-v2', // Lightweight, 2ms latency
+ });
+
+ return reranked.slice(0, topN);
+ }
+ ```
+
+ ---
+
### Implementation Checklist
- - [ ] Enable `pgvector` extension in PostgreSQL and create an `hnsw` index on the embedding column.
- - [ ] Implement Semantic Chunking (breaking documents by semantic boundaries rather than fixed character lengths).
- - [ ] Combine Vector Cosine Similarity with Full Text Search (BM25) using a weighted score (Hybrid Search).
- - [ ] Generate Hypothetical Document Embeddings (HyDE) to improve retrieval recall.
+ - [ ] Create `HNSW` index in PostgreSQL: `CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);`
+ - [ ] Configure `hnsw.ef_search = 100` for high-recall queries during production traffic.
+ - [ ] Implement Reciprocal Rank Fusion (RRF) with constant `k = 60` instead of arbitrary linear weighting.
+ - [ ] Add a Cross-Encoder Re-ranker step before injecting retrieved chunks into the LLM system prompt.
+ - [ ] Apply Late Chunking or Contextual Chunking to retain parent document continuity.
## Orchestration & Integration
- - Integrates with: `ai-llm-integration-expert`, `database-orm-expert`, `app-analyzer-optimizer`.
+ - Integrates with: `ai-llm-integration-expert`, `database-orm-expert`, `ai-cost-token-optimizer`, `app-analyzer-optimizer`.
---
<a name="bahasa-indonesia"></a>
## Bahasa Indonesia
### Deskripsi
- Panduan tingkat produksi untuk Vector Database (pgvector, Qdrant, Pinecone, Milvus), arsitektur RAG, indeks pgvector HNSW, hybrid search (dense + BM25 sparse re-ranking), semantic chunking, dan evaluasi RAG.
+ Panduan arsitektur tingkat produksi untuk Vector Database (PostgreSQL `pgvector 0.8+`, Qdrant, LanceDB, Pinecone), arsitektur Deep RAG modern, pencarian HNSW iteratif, **Reciprocal Rank Fusion (RRF)** hybrid retrieval, **Cross-Encoder Re-ranking** (Cohere Rerank v3, FlashRank, BGE-Reranker-v2), dan **Late Chunking** untuk mencegah fragmentasi konteks.
### Fitur Utama
- - **pgvector & Hybrid Search**: PostgreSQL `pgvector` HNSW indexing, tuning jarak cosine/L2, dan re-ranking BM25.
- - **Arsitektur RAG**: Retrieval Parent-Document, HyDE (Hypothetical Document Embeddings), dan kompresi kontekstual.
- - **Evaluasi RAG**: Scoring kualitas retrieval otomatis menggunakan Ragas dan TruLens.
+ 1. **pgvector 0.8+ & Indeks HNSW**: Penyimpanan vektor dimensi tinggi, tuning metrik jarak (cosine/inner-product/L2), dan pemindaian HNSW iteratif dengan filter metadata.
+ 2. **Reciprocal Rank Fusion (RRF)**: Menggabungkan peringkat kata kunci BM25 dengan peringkat semantik vektor menggunakan rumus $RRF(d) = \sum \frac{1}{k + rank(d)}$, jauh lebih akurat daripada pembobotan linear biasa.
+ 3. **Cross-Encoder Re-ranking**: Pipeline retrieval 2 tahap: ambil 50 kandidat teratas melalui pencarian hybrid, lalu urutkan ulang menjadi 5 dokumen paling relevan menggunakan model cross-encoder.
+ 4. **Late Chunking**: Melakukan embedding dokumen secara utuh dalam transformer sebelum memecahnya menjadi chunk-chunk terpisah, mempertahankan makna global dokumen.
+ 5. **Evaluasi RAG**: Pengukuran presisi retrieval dan deteksi halusinasi secara otomatis (Ragas, TruLens, DeepEval).
+ ---
+
+ ### Resep Implementasi Produksi
+
+ #### Resep 1: Pencarian Hybrid RRF dengan Drizzle ORM
+ ```typescript
+ import { sql } from 'drizzle-orm';
+ import { db } from '@/lib/db';
+
+ export async function cariDokumenRRF(
+ queryVector: number[],
+ queryText: string,
+ limit = 10,
+ k = 60
+ ) {
+ const vectorStr = JSON.stringify(queryVector);
+
+ const hasil = await db.execute(sql`
+ WITH vector_matches AS (
+ SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> ${vectorStr}::vector) AS rank
+ FROM documents
+ WHERE status = 'published'
+ ORDER BY embedding <=> ${vectorStr}::vector
+ LIMIT 50
+ ),
+ text_matches AS (
+ SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(fts, websearch_to_tsquery('english', ${queryText})) DESC) AS rank
+ FROM documents
+ WHERE fts @@ websearch_to_tsquery('english', ${queryText})
+ LIMIT 50
+ )
+ SELECT
+ d.id,
+ d.content,
+ COALESCE(1.0 / (${k} + v.rank), 0.0) +
+ COALESCE(1.0 / (${k} + t.rank), 0.0) AS skor_rrf
+ FROM documents d
+ LEFT JOIN vector_matches v ON d.id = v.id
+ LEFT JOIN text_matches t ON d.id = t.id
+ WHERE v.id IS NOT NULL OR t.id IS NOT NULL
+ ORDER BY skor_rrf DESC
+ LIMIT ${limit};
+ `);
+
+ return hasil.rows;
+ }
+ ```
+
+ #### Resep 2: Pipeline Re-Ranking dengan FlashRank (Node.js / TypeScript)
+ ```typescript
+ import { FlashRankRegistry } from 'flashrank';
+
+ const ranker = new FlashRankRegistry();
+
+ export async function susunUlangKandidat(kueri: string, kandidat: { id: string; text: string }[], topN = 5) {
+ const passages = kandidat.map(c => ({ id: c.id, text: c.text }));
+
+ const hasilRerank = await ranker.rerank({
+ query: kueri,
+ passages,
+ model: 'ms-marco-TinyBERT-L-2-v2',
+ });
+
+ return hasilRerank.slice(0, topN);
+ }
+ ```
+
+ ---
+
### Checklist Implementasi
- - [ ] Aktifkan ekstensi `pgvector` di PostgreSQL dan buat indeks `hnsw` pada kolom embedding.
- - [ ] Terapkan Semantic Chunking (memecah dokumen berdasarkan batas semantik alih-alih panjang karakter tetap).
- - [ ] Gabungkan Vector Cosine Similarity dengan Full Text Search (BM25) menggunakan skor berbobot (Hybrid Search).
- - [ ] Hasilkan Hypothetical Document Embeddings (HyDE) untuk meningkatkan recall retrieval.
+ - [ ] Buat indeks `HNSW` di PostgreSQL: `CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);`
+ - [ ] Konfigurasikan `hnsw.ef_search = 100` untuk kueri dengan recall tinggi di lingkungan produksi.
+ - [ ] Terapkan Reciprocal Rank Fusion (RRF) dengan konstanta `k = 60` alih-alih pembobotan linear manual.
+ - [ ] Tambahkan langkah Cross-Encoder Re-ranker sebelum menyuntikkan konteks ke prompt LLM.
+ - [ ] Terapkan Late Chunking agar konteks dokumen utuh tidak hilang saat dipotong.
## Integrasi Orkestrasi
- - Terintegrasi dengan: `ai-llm-integration-expert`, `database-orm-expert`, `app-analyzer-optimizer`.
+ - Terintegrasi dengan: `ai-llm-integration-expert`, `database-orm-expert`, `ai-cost-token-optimizer`, `app-analyzer-optimizer`.