kb-review · diff
git:20260902.281273b to git:20260902.a5d5e32
342 added, 63 removed. Audit A to A.
---
name: kb-review
description: |
Read-only review of a retrieval-augmented setup, Amazon Bedrock Knowledge Bases first and hand-built RAG pipelines second: sources, chunking, metadata and tenant filtering, embedding model, vector store, sync, retrieval quality and cost per query. Builds a golden question set, measures recall@5 and MRR against the live knowledge base, and leaves the eval set behind as JSONL. Use when someone asks whether the knowledge base is set up right, why answers miss the right document, or what a RAG pipeline costs to run.
triggers:
- knowledge base review
- rag review
- bedrock knowledge base
- chunking strategy
- retrieval quality
- recall at k
- vector store review
allowed-tools:
- Bash
- Read
- Grep
- Write
- AskUserQuestion
---
## When to invoke
Use when: "knowledge base review", "rag review", "bedrock knowledge base", "chunking strategy", "retrieval quality", "recall at k", "vector store review", "why does the bot not find the document".
## Preamble
```bash
eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
_LEARN_FILE="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}/learnings.jsonl"
if [ -f "$_LEARN_FILE" ]; then
_LEARN_COUNT=$(wc -l < "$_LEARN_FILE" 2>/dev/null | tr -d ' ')
echo "LEARNINGS: $_LEARN_COUNT entries loaded"
if [ "$_LEARN_COUNT" -gt 5 ] 2>/dev/null; then
~/.vibestack/bin/vibe-learnings-search --limit 5 2>/dev/null || true
fi
else
echo "LEARNINGS: none yet"
fi
```
{{include lib/snippets/session-host.md}}
{{include lib/snippets/decision-brief.md}}
{{include lib/snippets/working-protocols.md}}
{{include lib/snippets/state-protocols.md}}
## User-invocable
When the user types `/kb-review`, run this skill.
---
## Step 1: Inventory
Find out what the knowledge base is made of before judging any of it. Two places to
look: infrastructure code in the repo, and the live account.
**Repo first.** Grep for the Terraform resources and for the SDK calls a hand-built
pipeline would use:
```bash
grep -rnE 'aws_bedrockagent_(knowledge_base|data_source)|aws_opensearchserverless_collection|pgvector|s3_vectors|S3_VECTORS' --include='*.tf' --include='*.py' --include='*.ts' --include='*.yaml' --include='*.yml' . 2>/dev/null | head -40
grep -rnE 'RecursiveCharacterTextSplitter|SentenceSplitter|chunk_size|chunk_overlap|embed_documents|Pinecone|Qdrant|Weaviate|pgvector' --include='*.py' --include='*.ts' . 2>/dev/null | head -40
```
If the first grep hits, this is a Bedrock Knowledge Base and the Terraform is the
source of truth for configuration. If only the second hits, it is a hand-built
pipeline: read the ingestion and query code and map each of the following steps
onto it. If neither hits, ask where the knowledge base lives.
**Live account, when the CLI works.** Check credentials once and stop with a setup
note if they fail (profile tried, error text, `aws configure` or `aws sso login`).
Then list what exists:
```bash
aws sts get-caller-identity --output json
aws bedrock-agent list-knowledge-bases --output json
aws bedrock-agent get-knowledge-base --knowledge-base-id <kb-id> --output json
aws bedrock-agent list-data-sources --knowledge-base-id <kb-id> --output json
aws bedrock-agent get-data-source --knowledge-base-id <kb-id> --data-source-id <data-source-id> --output json
```
Every angle-bracket value in this skill — `<kb-id>`, `<data-source-id>`, `<bucket>`,
`<prefix>` and the rest — is substituted by hand from the output of the preceding
command before the line is run. They are placeholders, not shell variables.
- From these, fill in the KB summary table: knowledge base id and name; vector store
- type from `storageConfiguration.type` (OpenSearch Serverless, Aurora with pgvector,
- S3 Vectors, Pinecone, or another); embedding model ARN and dimensions from
- `knowledgeBaseConfiguration.vectorKnowledgeBaseConfiguration`; and per data source
- its type (S3, Confluence, SharePoint, web crawler, custom), location, chunking
- configuration, parsing configuration and `dataDeletionPolicy`.
+ **Read the knowledge base type first.** `knowledgeBaseConfiguration.type` is one of
+ `VECTOR`, `MANAGED`, `KENDRA` or `SQL`, and it decides which fields exist and which
+ retrieval request shape is valid in Steps 3 and 6:
+ - `VECTOR` — a knowledge base backed by a vector store you own. Embedding model ARN
+ and dimensions come from `knowledgeBaseConfiguration.vectorKnowledgeBaseConfiguration`;
+ the store comes from `storageConfiguration.type` (OpenSearch Serverless, Aurora
+ with pgvector, S3 Vectors, Pinecone, or another). Query with
+ `retrievalConfiguration.vectorSearchConfiguration`.
+ - `MANAGED` — Amazon Bedrock owns the vector store. Read
+ `knowledgeBaseConfiguration.managedKnowledgeBaseConfiguration`: `embeddingModelType`
+ is `MANAGED` (service-chosen model) or `CUSTOM` (with `embeddingModelArn`).
+ `storageConfiguration` is optional on the KnowledgeBase object and is absent here —
+ record the store as service-managed rather than reporting a missing field. Query
+ with `retrievalConfiguration.managedSearchConfiguration`.
+ - `KENDRA` — an Amazon Kendra GenAI index holds the content, so there is no chunking,
+ embedding or vector-store configuration to review. Query it with `Retrieve` as you
+ would a standard knowledge base, `vectorSearchConfiguration` and all; the Kendra
+ fields come back as metadata attributes and metadata filtering works on them. Say so,
+ skip Steps 2, 4 and 7's store item, and run Steps 3, 5 and 6 unchanged.
+ - `SQL` — a structured data store answers by generated SQL, not by embeddings, so there
+ is nothing chunked, embedded or stored as vectors to review. `Retrieve` returns the
+ rows the generated query produced, and none of `retrievalConfiguration` applies:
+ `numberOfResults`, search type, metadata filtering and reranking are all documented as
+ applying to unstructured data sources only. Send the query on its own, with no
+ `--retrieval-configuration` at all. Skip Steps 2, 4 and 7's store item. Step 3 keeps
+ its question but changes its evidence — there is no metadata filter to check, so read
+ the query engine's table and column inclusions and exclusions and the database grants
+ on the role the knowledge base queries with. Steps 5 and 6 run with the row-based
+ reading noted in each.
+
+ Sending a `vectorSearchConfiguration` to a managed knowledge base, or a
+ `managedSearchConfiguration` to a `VECTOR` one, is an invalid request — a failed call
+ there is a mistake in this review, not a finding about the knowledge base. So is sending
+ either one to a `SQL` knowledge base. Set the key once and reuse it:
+
+ ```bash
+ SEARCH_KEY=vectorSearchConfiguration # managedSearchConfiguration on a MANAGED KB
+ # SQL: send no retrieval configuration at all
+ ```
+
+ Then fill in the KB summary table: knowledge base id, name and type; the store and
+ embedding model from whichever branch above applies; and per data source its type
+ (S3, Confluence, SharePoint, OneDrive, Google Drive, Salesforce, Box, web crawler,
+ custom — record whatever `dataSourceConfiguration.type` reports, the connector set
+ grows), location, chunking configuration, parsing configuration and
+ `dataDeletionPolicy`.
+
**Sources.** For every S3 data source, size it without downloading it:
```bash
aws s3 ls s3://<bucket>/<prefix> --recursive --summarize | tail -2
aws s3 ls s3://<bucket>/<prefix> --recursive | awk '{print $4}' | sed -E 's/.*\.([A-Za-z0-9]+)$/\1/' | sort | uniq -c | sort -rn
```
Record object count, total size, and the format mix (PDF, DOCX, HTML, Markdown, CSV,
- JSON). Note the update cadence from the newest and oldest `LastModified` values and
+ JSON). Count the image objects (`.jpeg`, `.png`) on their own line: Step 7 prices a
+ configured parser per page and per image, and neither number falls out of the object
+ count. When any data source sets a `parsingConfiguration`, carry a page estimate into
+ Step 7 as well — read the page count of each PDF sampled in Step 2 (`pdfinfo`, or
+ `mdls -name kMDItemNumberOfPages` on macOS), take the mean, multiply by the PDF object
+ count, and record the sample size beside it so the number is read as the estimate it
+ is. Note the update cadence from the newest and oldest `LastModified` values and
from whatever writes to the prefix (a pipeline, a person, a Confluence export). For
Confluence and web sources, record the space keys or seed URLs and the crawl scope.
Findings at this step: **INFO** for every data source, **MEDIUM** if a data source
points at a bucket prefix that also holds unrelated files (logs, exports, backups)
that will be embedded along with the documents.
---
## Step 2: Chunking
Read `vectorIngestionConfiguration.chunkingConfiguration` from each data source (or
the splitter call in a hand-built pipeline) and compare it against the shape of the
documents. Sample three to five documents per format into the scratchpad with
`aws s3 cp`; look at them, do not paste them into the report.
Check:
- - **Strategy versus document shape.** Fixed-size chunks suit short, flat documents. Hierarchical chunking (parent and child levels) suits long manuals and policies where a paragraph only makes sense with its section heading. Semantic chunking suits prose with topic shifts. `NONE` means each document is one chunk, which only works for documents already under the embedding model's token limit. **HIGH** if strategy is `NONE` and any sampled document exceeds that limit: those documents are silently truncated at ingestion.
+ - **Strategy versus document shape.** Fixed-size chunks suit short, flat documents. Hierarchical chunking (parent and child levels) suits long manuals and policies where a paragraph only makes sense with its section heading. Semantic chunking suits prose with topic shifts. `NONE` means each document is treated as a single source chunk; the service applies its own chunk handling from there, and a document that violates a limit surfaces in the ingestion job, not as a quiet truncation. Do not infer truncation from the embedding model's input limit — check it. Read `numberOfDocumentsFailed` and `failureReasons` on the last job (Step 5), then run one retrieve for a phrase that appears only near the end of the largest sampled document and read `content.text` on the results. **HIGH** if documents failed ingestion, or if that tail phrase is in no returned chunk while the same document answers other questions. **MEDIUM** if the strategy is `NONE` on long documents and neither check could be run — the risk is real, the evidence is not there yet. Note also that `NONE` gives up page-number citations and the `x-amz-bedrock-kb-document-page-number` filter attribute.
- **Chunk size and overlap.** Read `maxTokens` and `overlapPercentage` (fixed) or the level `maxTokens` and `overlapTokens` (hierarchical). **MEDIUM** if chunks are under 200 tokens on documents with long paragraphs (answers get split across chunks) or over 1000 tokens on FAQ-style content (one chunk carries several unrelated questions). **LOW** if overlap is 0 on fixed-size chunking.
- **Tables and code.** Open a sampled PDF or DOCX that contains a table and find the corresponding chunk text with a `retrieve` call for a value from the table. **MEDIUM** if table rows come back as a run of numbers with no header context and no parsing configuration is set (`parsingConfiguration` with a foundation model or Bedrock Data Automation keeps table structure). Same check for code blocks in Markdown.
- **Metadata files.** Each document may carry a sidecar `<name>.metadata.json` with a `metadataAttributes` object. Count them: `aws s3 ls s3://<bucket>/<prefix> --recursive | grep -c '\.metadata\.json$'`. Compare with the document count. **MEDIUM** if fewer than 90% of documents have one and Step 3 relies on filtering.
---
## Step 3: Metadata and isolation
This is the step that decides whether one customer can read another customer's
documents. Treat every gap here as at least **HIGH**.
- **What is in the metadata.** Read three sampled `.metadata.json` files. The attributes that matter are tenant (or customer, account), department, and classification (public, internal, confidential). Record which exist and whether values are consistent (`tenant` versus `tenantId` versus `customer` across files is a finding).
- - **Filter applied on every retrieve.** Find every call site of `retrieve`, `retrieve_and_generate`, `RetrieveCommand`, or the agent's knowledge base configuration, and check that `retrievalConfiguration.vectorSearchConfiguration.filter` is set from the caller's identity, not from a request parameter the caller controls. `grep -rnE 'retrieve(_and_generate|AndGenerate)?\(|RetrieveCommand|RetrieveAndGenerateCommand' --include='*.py' --include='*.ts' --include='*.js' .` **CRITICAL** if any call site queries the knowledge base with no filter while the knowledge base holds more than one tenant. **HIGH** if the filter value comes from the request body or a query parameter.
- - **Prove the leakage path is closed.** Run one retrieve with a filter for tenant A and a question that only tenant B's documents answer:
+ - **Which mechanism is in play.** Two are supported and they are checked differently, so decide before judging anything. *Metadata filtering*: documents carry a tenant attribute and every query passes a `filter`. *ACL awareness*, on a managed knowledge base: the connector crawls document permissions (allowed and denied users and groups) at ingestion, and the query passes a `userContext` naming the user, and the crawled permissions filter the candidates before retrieval. Some connectors add a second stage, a live call to the source per query that catches permission changes made since the last sync. Which ones do is a per-connector property and the connector set grows, so read it from that connector's own ACL page rather than from a list held in your head. Today: SharePoint, OneDrive, Google Drive, Confluence and Box document both stages; S3 and custom document pre-retrieval filtering only, because their ACLs come from a file the customer writes and there is nothing live to re-check; the web crawler supports no ACLs at all. For any connector you have not checked, record its real-time support as unknown and go read the page before a severity leans on it. Read each data source's configuration for ACL awareness, and read the call sites for which of `filter` and `userContext` they send. A knowledge base may use one, both, or neither. An ACL-enabled data source returning nothing when `userContext` is absent is documented behaviour, not a retrieval bug; non-ACL data sources in the same knowledge base still return to everyone.
+ - **Filter applied on every retrieve.** Find every call site of `retrieve`, `retrieve_and_generate`, `RetrieveCommand`, or the agent's knowledge base configuration: `grep -rnE 'retrieve(_and_generate|AndGenerate)?\(|RetrieveCommand|RetrieveAndGenerateCommand' --include='*.py' --include='*.ts' --include='*.js' .` On a metadata-filtered knowledge base, check that `retrievalConfiguration.$SEARCH_KEY.filter` is set from the caller's authenticated session, not from a request parameter the caller controls. **CRITICAL** if a call site queries a multi-tenant knowledge base with neither a filter nor a user context. **HIGH** if the filter value comes from the request body, a header or a query parameter.
+ - **User context derived from a verified identity.** On an ACL-aware knowledge base the same question applies to `userContext.userId`. The service does not authenticate end users and cannot verify the identity it is handed — it filters on whatever you send — so that email must come from the verified session (the token subject or an ID-token claim). **CRITICAL** if `userId` is caller-supplied: one edited request reads another tenant's documents. **HIGH** if the address passed is not the one the connected source knows the user by; matching is on the email with no alias resolution, so a mismatch returns nothing and invites someone to "fix" it by dropping the context. **MEDIUM** if group membership is the only thing standing between tenants *and* the connector behind that data source has no real-time check — memberships are resolved from the last crawl, so a removal takes effect only at the next sync. On a connector that re-verifies per query the live check corrects that at query time: record it **INFO**. If which of the two the connector is was never established, say the real-time support is unknown and leave the severity off rather than guessing at one.
+ - **Prove the leakage path is closed.** Run one retrieve as tenant A with a question only tenant B's documents answer. Substitute `<kb-id>` and use the branch that matches the mechanism:
```bash
- aws bedrock-agent-runtime retrieve --knowledge-base-id <kb-id> \
+ eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
+ EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
+ mkdir -p "$EVAL_DIR"
+ OUT="$EVAL_DIR/kb-isolation-$(date +%Y-%m-%d).json"
+ SEARCH_KEY=vectorSearchConfiguration # managedSearchConfiguration on a MANAGED KB
+ CFG=$(jq -nc --arg k "$SEARCH_KEY" '{($k):{numberOfResults:5,filter:{equals:{key:"tenant",value:"tenant-a"}}}}')
+ ERR=$(aws bedrock-agent-runtime retrieve --knowledge-base-id <kb-id> \
--retrieval-query "$(jq -nc --arg text '<question only tenant B can answer>' '{text:$text}')" \
- --retrieval-configuration '{"vectorSearchConfiguration":{"numberOfResults":5,"filter":{"equals":{"key":"tenant","value":"tenant-a"}}}}' \
- --output json | grep -E '"uri"|"score"'
+ --retrieval-configuration "$CFG" \
+ --output json 2>&1 > "$OUT")
+ rc=$?
+ if [ "$rc" -ne 0 ]; then
+ printf 'ISOLATION: N-A (exit %s)\n%s\n' "$rc" "$ERR"
+ else
+ jq -r '.retrievalResults[] | [(.location.type // "?"), (.documentId // "-"), (.score|tostring)] | @tsv' "$OUT"
+ printf 'ISOLATION: %s result(s)\n' "$(jq '.retrievalResults | length' "$OUT")"
+ fi
```
+ On an ACL-aware knowledge base, drop the `filter` from `CFG` and add
+ `--user-context '{"userId":"user-a@example.com"}'` with an address that belongs to
+ tenant A.
+
+ On a `SQL` knowledge base there is no filter and no user context to send: drop
+ `--retrieval-configuration` and the `CFG` line with it, run the probe as the caller the
+ application would use for tenant A, and read whether the returned rows belong to tenant
+ A. What holds tenants apart there is the query engine's table and column inclusions and
+ exclusions plus the grants on the database role, so review those alongside the probe —
+ a generated query that can reach another tenant's rows is **CRITICAL** on the same
+ terms as a missing filter.
+
Pass the query as JSON, not as the `text=...` shorthand: the shorthand parser splits
on commas, so any question containing one is read as a list and the call fails with
`Invalid type for parameter retrievalQuery.text`. That kills German questions in
particular. Without `jq`, write the object out by hand:
`--retrieval-query '{"text":"Wie lange ist die Frist, wenn ich kuendige?"}'`.
- Every returned `uri` must belong to tenant A. A tenant B document in the result is **CRITICAL**, and the fix is at ingestion (missing or wrong metadata), not in the query.
+ Read the exit status before the results. An empty list is a pass only when `rc` is 0;
+ a denied `bedrock:Retrieve`, a throttle and a genuinely empty result all print
+ nothing, and only the status tells them apart, so report N-A on a non-zero exit
+ rather than a clean bill. On a zero exit, every returned document must belong to
+ tenant A. A tenant B document is **CRITICAL**, and the fix is at ingestion (missing or
+ wrong metadata, or permissions the connector did not crawl), not in the query.
- **Source access control mirrored.** If the S3 bucket, Confluence space, or SharePoint site restricts who can read what, that restriction has to be reproduced as metadata plus a filter, because the knowledge base reads everything with its service role. **HIGH** if a restricted source feeds the knowledge base and no metadata carries the restriction.
---
## Step 4: Embeddings
- **Model and dimension.** From Step 1, record the embedding model id (for example `amazon.titan-embed-text-v2:0` or `cohere.embed-multilingual-v3`) and the configured dimensions. Confirm the vector index in the store was created with the same dimension: for OpenSearch Serverless read the index mapping, for pgvector read the column type (`vector(1024)`). A mismatch fails at ingestion and shows up as every job failing; **HIGH**.
- **Language coverage.** Ask which languages the documents and the users are in. The usual case is German plus English. **HIGH** if the corpus or the questions include German and the model is English-only (`cohere.embed-english-v3`). Titan Text Embeddings v2 and Cohere Embed Multilingual cover both. Verify with the golden set in Step 6: include at least five German questions against English passages and five the other way round, and report their recall separately.
- **Re-embedding plan.** An embedding model cannot be changed on an existing Bedrock knowledge base; a model change means a new knowledge base, a full ingestion, and a cutover. **MEDIUM** if nothing in the repo or runbooks describes how that is done, and note the full ingestion cost from Step 7 next to the finding.
---
## Step 5: Sync
The knowledge base only knows what the last ingestion job saw.
```bash
aws bedrock-agent list-ingestion-jobs --knowledge-base-id <kb-id> --data-source-id <data-source-id> --max-results 20 --output json
```
For each data source, record the last job's status, start time, and its
`statistics` (documents scanned, new, modified, deleted, failed) plus
`failureReasons` when present. Then check:
- **Schedule.** Find what starts the job: an EventBridge rule, a Step Functions state machine, a CI step, or a person. `grep -rnE 'start_ingestion_job|StartIngestionJob|start-ingestion-job' .` and `aws events list-rules --output json | grep -i ingest`. **MEDIUM** if the only trigger is manual and the source changes weekly or faster. **HIGH** if the last successful job is older than two update cycles of the source.
- **Failure alerts.** **MEDIUM** if a job can fail with nobody told; look for a CloudWatch alarm or a notification on the job status. `numberOfDocumentsFailed` above zero on the last job with no alert is the same finding.
- **Deletions propagate.** `dataDeletionPolicy` on the data source: `DELETE` removes vectors when a document leaves the source, `RETAIN` keeps them. **HIGH** if `RETAIN` is set and documents are ever removed for legal or contractual reasons; the knowledge base will keep answering from them.
- **Source versioning.** **LOW** if the S3 bucket has versioning off and nobody can say what the knowledge base was built from a month ago. **INFO** if it is on.
For a hand-built pipeline, the same four questions apply to whatever loop upserts
into the vector store.
---
## Step 6: Retrieval quality
Numbers, not opinions. Build a golden set, run retrieve-only against it, then check a
sample of generated answers.
**Golden set.** Write 20 to 40 question and expected-passage pairs. Draw questions
from the sampled documents in Step 2, from support tickets or chat logs if the user
can share them, and from the user directly. Each line names the source document that
answers it and a short quote of the passage. Include the language split from Step 4
and at least three questions that the corpus does not answer (the right result is
nothing above the score threshold). Write it here:
```bash
eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
mkdir -p "$EVAL_DIR"
EVAL_SET="$EVAL_DIR/kb-eval-$(date +%Y-%m-%d).jsonl"
echo "EVAL_SET: $EVAL_SET"
```
One JSON object per line:
```json
{"id":"q01","question":"Wie lange ist die Kündigungsfrist?","lang":"de","tenant":"tenant-a","expected_uri":"s3://docs/tenant-a/contract-terms.pdf","expected_passage":"Die Kündigungsfrist beträgt drei Monate zum Quartalsende.","answerable":true}
```
The `tenant` field belongs there only when Step 3 found a tenant attribute in the
document metadata. On a knowledge base with no such attribute, leave it out — every
line then carries the same fields minus `tenant`.
**Retrieve-only run.** For every line, call `retrieve` with `numberOfResults` 5,
- filtered by tenant when the line has one, and record the ranked list of `uri` values
- and scores. The filter clause is dropped for a line with no `tenant` field: filtering
- on a metadata key the documents do not carry returns nothing for every question, which
- scores recall@5 at 0.00 and manufactures a HIGH finding against a knowledge base that
- is fine. With `jq` present the loop is this, with `<kb-id>` substituted before it runs
- (it re-derives the eval-set path, since each block runs in its own shell):
+ filtered by tenant when the line has one, and record the ranked hits: the document
+ the hit came from, its `documentId`, its `content.type`, its `content.text`, its
+ `content.row` and its score. Keep the text — the metrics below cannot be computed
+ without it — and keep the row with it: a structured result carries its payload in
+ `content.row`, an array of `columnName`, `columnValue` and `type`, with `content.type`
+ set to `ROW` and `content.text` empty. `documentId` identifies the
+ document, not the chunk, and a retrieval result carries no per-chunk identifier, so
+ the passage judgement below is made from the text and nothing else can stand in for
+ it. The filter clause is dropped for a
+ line with no `tenant` field: filtering on a metadata key the documents do not carry
+ returns nothing for every question, which scores 0.00 and manufactures a HIGH finding
+ against a knowledge base that is fine.
+ Result locations come in ten shapes, one per source type, and only one is populated
+ per hit; `location` can also be absent. Reading three of them and calling the rest
+ null turns every Salesforce, SharePoint, OneDrive, Google Drive, Kendra, custom or
+ structured hit into a miss. Normalise all of them, and fall back to `documentId`.
+
+ With `jq` present the loop is this, with `<kb-id>` substituted before it runs (it
+ re-derives the eval-set path, since each block runs in its own shell):
+
```bash
+ set -o pipefail
eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
EVAL_SET="$EVAL_DIR/kb-eval-$(date +%Y-%m-%d).jsonl"
RESULTS="$EVAL_DIR/kb-eval-results-$(date +%Y-%m-%d).jsonl"
+ SEARCH_KEY=vectorSearchConfiguration # managedSearchConfiguration on a MANAGED KB
: > "$RESULTS"
while IFS= read -r line; do
+ id=$(printf '%s' "$line" | jq -r .id)
q=$(printf '%s' "$line" | jq -r .question)
t=$(printf '%s' "$line" | jq -r '.tenant // empty')
if [ -n "$t" ]; then
- cfg=$(jq -nc --arg t "$t" '{vectorSearchConfiguration:{numberOfResults:5,filter:{equals:{key:"tenant",value:$t}}}}')
+ cfg=$(jq -nc --arg k "$SEARCH_KEY" --arg t "$t" '{($k):{numberOfResults:5,filter:{equals:{key:"tenant",value:$t}}}}')
else
- cfg='{"vectorSearchConfiguration":{"numberOfResults":5}}'
+ cfg=$(jq -nc --arg k "$SEARCH_KEY" '{($k):{numberOfResults:5}}')
fi
- aws bedrock-agent-runtime retrieve --knowledge-base-id <kb-id> \
+ raw=$(aws bedrock-agent-runtime retrieve --knowledge-base-id <kb-id> \
--retrieval-query "$(jq -nc --arg text "$q" '{text:$text}')" \
--retrieval-configuration "$cfg" \
- --output json \
- | jq -c --argjson g "$line" '{id:$g.id, expected:$g.expected_uri, hits:[.retrievalResults[] | {uri:(.location.s3Location.uri // .location.webLocation.url // .location.confluenceLocation.url), score:.score}]}' >> "$RESULTS"
+ --output json 2>&1)
+ rc=$?
+ if [ "$rc" -ne 0 ]; then
+ jq -nc --arg id "$id" --argjson rc "$rc" --arg err "$raw" \
+ '{id:$id, status:"error", exit:$rc, error:$err, hits:[]}' >> "$RESULTS"
+ continue
+ fi
+ printf '%s' "$raw" | jq -c --argjson g "$line" '
+ {id:$g.id, status:"ok", expected:$g.expected_uri,
+ hits:[.retrievalResults[] | {
+ doc: (.location.s3Location.uri // .location.webLocation.url
+ // .location.confluenceLocation.url // .location.salesforceLocation.url
+ // .location.sharePointLocation.url // .location.oneDriveLocation.url
+ // .location.googleDriveLocation.url // .location.kendraDocumentLocation.uri
+ // .location.customDocumentLocation.id // .location.sqlLocation.query
+ // .documentId // null),
+ loc_type: (.location.type // "UNKNOWN"),
+ doc_id: (.documentId // null),
+ content_type: (.content.type // "UNKNOWN"),
+ text: (.content.text // ""),
+ row: (.content.row // []),
+ score: .score}]}' >> "$RESULTS"
done < "$EVAL_SET"
```
Same reason for the query object as in Step 3: `text=$q` is shorthand and splits on
the first comma, so a question with a comma in it fails the call and silently drops
that line from the score.
+ On a `SQL` knowledge base drop `--retrieval-configuration` and both `cfg` branches —
+ the settings they carry apply to unstructured sources only — and judge each answer
+ against the expected value the golden line names rather than against a passage. The
+ judgement reads `row`, not `text` — `text` is empty on every structured hit, and the
+ returned value is the `columnValue` of the `row` entry whose `columnName` is the one
+ the golden line expects. There are no chunks to rank, so report the share of questions
+ answered with the right value and leave the passage and MRR columns blank.
+
+ **Check the denominator before scoring anything.** A throttle or a denied
+ `bedrock:Retrieve` costs one row, and a metric computed over the survivors reads
+ better than the truth:
+
+ This block runs in its own shell, so it re-derives the paths the same way Steps 3, 6
+ and 7 do. It exits non-zero on anything it cannot establish — an unwritable id file
+ prints an all-clear otherwise, which is the failure it exists to catch:
+
+ ```bash
+ set -euo pipefail
+ eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
+ EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
+ EVAL_SET="$EVAL_DIR/kb-eval-$(date +%Y-%m-%d).jsonl"
+ RESULTS="$EVAL_DIR/kb-eval-results-$(date +%Y-%m-%d).jsonl"
+ ASKED="$EVAL_DIR/.ids-asked"
+ GOT="$EVAL_DIR/.ids-got"
+ for f in "$EVAL_SET" "$RESULTS"; do
+ [ -s "$f" ] || { printf 'DENOMINATOR: N-A (missing or empty %s)\n' "$f"; exit 1; }
+ done
+ jq -r .id "$EVAL_SET" | sort > "$ASKED" || { printf 'DENOMINATOR: N-A (cannot write %s)\n' "$ASKED"; exit 1; }
+ jq -r .id "$RESULTS" | sort > "$GOT" || { printf 'DENOMINATOR: N-A (cannot write %s)\n' "$GOT"; exit 1; }
+ missing=$(comm -23 "$ASKED" "$GOT")
+ dupes=$(uniq -d < "$GOT")
+ errs=$(jq -r 'select(.status=="error") | .id' "$RESULTS" | tr '\n' ' ')
+ printf 'asked=%s got=%s\n' "$(wc -l < "$ASKED" | tr -d ' ')" "$(wc -l < "$GOT" | tr -d ' ')"
+ printf 'missing=[%s] duplicate=[%s] errors=[%s]\n' "$missing" "$dupes" "$errs"
+ [ -z "${missing}${dupes}${errs}" ] || { printf 'DENOMINATOR: FAIL — do not score this run\n'; exit 1; }
+ printf 'DENOMINATOR: OK\n'
+ ```
+
+ A non-zero exit, or anything but `DENOMINATOR: OK` on the last line, stops the
+ scoring. Fix the cause and re-run the affected
+ questions — pause and retry on a throttle, get the permission for an authorization
+ error — or, if it cannot be fixed, report the metrics as N-A for that subset and
+ state how many questions were dropped and why. Do not score a partial set as if it
+ were whole.
+
Without `jq`, run the calls one at a time and record the ranks by hand; the set is
small enough.
- **Metrics.** For the answerable questions: recall@5 is the share whose
- `expected_uri` appears anywhere in the five hits; MRR is the mean of 1/rank of the
- first hit (0 when absent). For the unanswerable questions, report how many returned a
- top score above the threshold the application uses (a false positive). Report each
- metric for the whole set and per language. Findings: **HIGH** if recall@5 is under
- 0.7, **MEDIUM** under 0.85, **LOW** if MRR is under 0.5 while recall@5 is fine (the
- right chunk is there but ranked low; a reranker or smaller chunks are the usual fix).
- **MEDIUM** if a language subset trails the other by more than 0.15.
+ **Metrics.** Two different measurements, and the difference decides what a number is
+ allowed to claim:
+ - **Document hit rate@5** — the share of answerable questions whose `expected_uri`
+ matches the `doc` of any of the five hits. It says the right file came back. It
+ says nothing about the passage: on a long manual, a chunk about an unrelated
+ section scores exactly as a hit, so this number alone cannot support a claim about
+ retrieval quality and cannot feed a rank-based metric.
+ - **Passage hit rate@5 and MRR** — read the `text` of the hits in rank order and mark
+ the first one that actually answers the question. `expected_passage` is the
+ reference; an equally correct passage elsewhere counts, a chunk from the right
+ document that does not answer it does not. Passage hit rate@5 is the share with
+ such a hit among the five; MRR is the mean of 1/rank of that hit, 0 when there is
+ none. Record the rank you assigned per question alongside its row so the next
+ review can see the judgement.
+
+ Report both, for the whole set and per language. If the passages were not judged,
+ print the document number under its own name, leave the passage and MRR columns
+ blank, and write "passage relevance not judged" under the table — do not carry the
+ document-level rank into the MRR column.
+
+ For the unanswerable questions, report how many returned a top score above the
+ threshold the application uses (a false positive).
+
+ Findings, all read off the passage numbers: **HIGH** if passage hit rate@5 is under
+ 0.7, **MEDIUM** under 0.85, **LOW** if MRR is under 0.5 while passage hit rate@5 is
+ fine (the right chunk is there but ranked low; a reranker or smaller chunks are the
+ usual fix). **MEDIUM** if a language subset trails the other by more than 0.15. When
+ only the document number was measured, cap every one of these at **MEDIUM** and say
+ so — that evidence cannot tell a right passage from a wrong one in the right file.
+
**Generation and faithfulness.** Pick 10 answerable questions and run
`aws bedrock-agent-runtime retrieve-and-generate` (or the application's own answer
endpoint) with citations on. For each answer check two things by reading the cited
passages: every factual claim in the answer appears in a cited passage, and the
citation points at the expected document. Score each answer `faithful`, `partly`
(a claim with no support), or `wrong`. **HIGH** if two or more of ten are `wrong`.
Record the model id and the prompt template used, since both move the result.
These calls cost tokens. Say the count before running (40 retrieves plus 10
generations is the default) and do not loop beyond the golden set.
---
## Step 7: Cost
- Four line items make up the bill. Pull current unit prices from the billing MCP
- pricing tool when the session has it, otherwise from the public pricing page for
- each service, and write the date the price was read next to it.
+ The bill has more parts than the embeddings. Walk the list, mark the items this
+ knowledge base actually uses, and drop the rest with a line saying it is not
+ configured. Pull current unit prices from the billing MCP pricing tool when the
+ session has it, otherwise from the public pricing page for each service, and write
+ the date the price was read next to it.
- 1. **Vector store.** OpenSearch Serverless bills OCU-hours with a floor (two OCUs for a collection with redundancy off, four with it on) plus GB-month of index storage; the floor dominates for small corpora. Aurora Serverless v2 bills ACU-hours plus storage. S3 Vectors bills storage plus per-query and per-put requests. Take the actual monthly figure from Cost Explorer when the account allows it: `aws ce get-cost-and-usage --time-period Start=<first of last month>,End=<first of this month> --granularity MONTHLY --metrics UnblendedCost --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon OpenSearch Service"]}}'`, adjusting the service name to the store in use.
- 2. **Embedding tokens at ingestion.** Total corpus tokens (roughly total bytes of extracted text divided by 4, times 1 plus the overlap fraction) times the embedding price per 1000 tokens. This is the cost of one full sync and of any re-embedding.
- 3. **Embedding tokens per query.** One query embedding per retrieve; small but nonzero.
- 4. **Generation tokens per query.** Prompt tokens are the template plus the five chunks (chunk `maxTokens` times 5) plus the question; output tokens from the Step 6 answers. Multiply by the model's input and output prices.
+ 1. **Vector store.** Read the live configuration; there is no fixed floor to assume.
+ OpenSearch Serverless capacity is a minimum and a maximum OCU count for indexing
+ and for search, set per collection group — or at account level for a Classic
+ collection that belongs to no group. The minimum can be 0, in which case an idle
+ group needs no OCUs at all, and collections inside a group share OCUs. So the idle
+ cost is whatever the configured minimums are, divided across every collection
+ sharing them, not two or four per collection. Read it and measure it:
+ ```bash
+ aws opensearchserverless batch-get-collection --names <collection-name> --output json
+ aws opensearchserverless list-collection-groups --output json
+ aws opensearchserverless batch-get-collection-group --ids <collection-group-id> --output json
+ aws opensearchserverless get-account-settings --output json
+ aws cloudwatch get-metric-statistics --namespace AWS/AOSS --metric-name SearchOCU \
+ --start-time <30 days ago> --end-time <now> --period 3600 \
+ --statistics Average Maximum --output json
+ ```
+
+ Record the configured minimums, the measured `SearchOCU` and `IndexingOCU`
+ averages, and how many collections share the group; repeat the metric call for
+ `IndexingOCU`. Aurora Serverless v2 bills ACU-hours plus storage — read the
+ cluster's minimum capacity and the `ServerlessDatabaseCapacity` metric the same
+ way. S3 Vectors bills storage plus requests, so it has no idle floor and its cost
+ tracks query and put volume.
+ 2. **Embedding tokens at ingestion.** Corpus tokens are roughly the bytes of
+ extracted text divided by 4. Overlap repeats text: with fixed-size chunking at
+ `maxTokens` M and `overlapPercentage` p as a fraction, consecutive chunks advance
+ by M(1-p), so the corpus is embedded about 1/(1-p) times over — 1.25x at 20%, 2x
+ at 50%. It is not 1+p, and the gap widens exactly where it costs most.
+ Hierarchical chunking embeds both levels, so count parent and child separately.
+ Multiply by the embedding price per 1,000 tokens. This is the cost of one full
+ sync and of any re-embedding.
+ 3. **Semantic chunking, when configured.** A `chunkingStrategy` of `SEMANTIC` calls a
+ foundation model during ingestion and is charged for that on top of the
+ embeddings, scaling with corpus size. Zero for every other strategy.
+ 4. **Parsing, when configured.** Neither parser is charged per document, so the object
+ count from Step 1 cannot price this line. `BEDROCK_DATA_AUTOMATION` is priced per
+ page of the document and per image processed: multiply the page estimate and the
+ image count recorded in Step 1 by the per-page and per-image rates.
+ `BEDROCK_FOUNDATION_MODEL` is priced on the input and output tokens the parser
+ model processes: estimate input tokens from the extracted text of the pages it
+ parses, output tokens from the extraction it returns, and multiply by that model's
+ input and output prices. On a large scanned-PDF corpus this can be the biggest
+ single ingestion line, and per-document pricing understates a 200-page PDF by two
+ orders of magnitude. Either parser, once set, is applied to every `.pdf` in the
+ data source, text-only ones included, so the whole PDF count is in scope. The
+ default parser adds nothing.
+ 5. **Embedding tokens per query.** One query embedding per retrieve; small but
+ nonzero.
+ 6. **Reranking, when configured.** A `rerankingConfiguration` on the search
+ configuration sends the candidate chunks through a reranker model on every query,
+ which is charged per query and can rival the generation cost on a chatty
+ application. Zero when `rerankingModelType` is `NONE` or no reranking is set.
+ 7. **Vector store request and data-processing charges.** Separate from item 1's
+ capacity or storage: S3 Vectors charges per query and per put, and the store's own
+ data-processing charges scale with query volume. On a high-volume, small-corpus
+ knowledge base these can exceed the embedding cost, which is why they cannot be
+ left out of a per-query figure.
+ 8. **Generation tokens per query.** Prompt tokens are the template plus the five
+ chunks (chunk `maxTokens` times 5) plus the question; output tokens from the Step 6
+ answers. Multiply by the model's input and output prices.
+
+ **Getting the store's real monthly figure.** Filter Cost Explorer down to this store.
+ A service-only filter returns the whole account's spend for that service, which on any
+ shared account is not this knowledge base's bill:
+
+ ```bash
+ eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
+ EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
+ mkdir -p "$EVAL_DIR"
+ CE_OUT="$EVAL_DIR/kb-cost-$(date +%Y-%m-%d).json"
+ CE_ERR=$(aws ce get-cost-and-usage \
+ --time-period Start=<first of last month>,End=<first of this month> \
+ --granularity MONTHLY --metrics UnblendedCost \
+ --filter '{"And":[{"Dimensions":{"Key":"SERVICE","Values":["Amazon OpenSearch Service"]}},{"Tags":{"Key":"<cost-allocation-tag>","Values":["<value>"]}}]}' \
+ --group-by Type=DIMENSION,Key=USAGE_TYPE \
+ --output json 2>&1 > "$CE_OUT")
+ rc=$?
+ [ "$rc" -ne 0 ] && printf 'COST: N-A (exit %s)\n%s\n' "$rc" "$CE_ERR"
+ ```
+
+ `--filter` takes `Dimensions`, `Tags` and `CostCategories` combined with
+ `And`, `Or` and `Not`, so swap the tag clause for a cost category when the account
+ uses one. The tag has to be activated as a cost allocation tag and only covers usage
+ from its activation date. Cost Explorer is a separate permission from the rest of this
+ review: on a non-zero exit, or when the store carries no tag or category that isolates
+ it, report the store cost as N-A and price it instead from the measured usage in item
+ 1 against published unit rates. Say in the report which of the two routes produced the
+ number. Never print an unfiltered service total as this store's cost.
+
Report a per-1000-queries figure: fixed monthly cost of the store divided by the
- observed or expected monthly query volume, plus items 3 and 4. Findings: **MEDIUM** if
- the store's idle floor is more than half of the monthly bill at the current query
- volume (S3 Vectors or pgvector on an existing database would cost less); **LOW** if
- prompt tokens per query are over 4000 because of large chunks times `numberOfResults`.
+ observed or expected monthly query volume, plus the per-query items. Findings:
+ **MEDIUM** if the store's idle floor is more than half of the monthly bill at the
+ current query volume (S3 Vectors or pgvector on an existing database would cost less)
+ — this one needs both the configured minimum capacity and a month of OCU or ACU
+ metrics showing usage sitting at that minimum; with only one of the two it is **LOW**
+ and written as a question for the user, not a verdict. **LOW** if prompt tokens per
+ query are over 4000 because of large chunks times `numberOfResults`.
---
## Output
```bash
eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
EVAL_DIR="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}"
mkdir -p "$EVAL_DIR"
REPORT="$EVAL_DIR/kb-review-$(date +%Y-%m-%d).md"
echo "REPORT: $REPORT"
```
Write the report to `$REPORT` (overwrite today's if it exists) and print it in full.
```
KNOWLEDGE BASE REVIEW
=====================
Account: <alias or last 4 digits> (profile: <name>) Region: <region>
Date: 2026-08-14
- Mode: Bedrock Knowledge Base | hand-built pipeline
+ Mode: Bedrock knowledge base (type VECTOR) | hand-built pipeline
KB SUMMARY
Field Value
----------------- ----------------------------------------------
- Knowledge base kb-docs-prod (ABCD1234)
- Vector store OpenSearch Serverless, 2 OCU floor, 1024-dim index
+ Knowledge base kb-docs-prod (ABCD1234), type VECTOR
+ Vector store OpenSearch Serverless, group min 2 index / 2 search OCU shared
+ with 3 collections, 1024-dim index
Embedding model amazon.titan-embed-text-v2:0, 1024 dims
Data sources s3://docs/tenant-*/ (1,240 PDF, 380 DOCX, 4.1 GB), confluence ENG space
Chunking hierarchical, parent 1500 / child 300 tokens, overlap 60
Parsing none (default text extraction)
+ Isolation metadata filter on tenant (no ACL awareness configured)
Metadata tenant, classification on 1,580 of 1,620 documents
Deletion policy RETAIN
Last sync 2026-08-12 03:10 UTC, 0 failed, EventBridge nightly
Languages de 70% / en 30%
FINDINGS
# Severity Area Evidence Fix
- -------- ---------- ---------------------------------------------------- --------------------------------------------
1 CRITICAL isolation api/search.py:88 retrieve() has no filter derive filter from the session tenant
2 HIGH sync dataDeletionPolicy RETAIN; legal deletes monthly set DELETE, re-run sync, verify vector count
3 MEDIUM chunking table values in pricing.pdf split across 3 chunks parsingConfiguration with a foundation model
4 MEDIUM metadata 40 documents without .metadata.json generate sidecars in the export job
- 5 LOW cost OCU floor is 68% of monthly spend at 9k queries evaluate S3 Vectors for this volume
+ 5 MEDIUM cost min 2 OCU idle = 68% of tagged spend; SearchOCU avg 2.0 evaluate S3 Vectors for this volume
- RETRIEVAL METRICS (golden set: 32 questions, 5 unanswerable)
- Subset n recall@5 MRR false positives
- ---------- --- -------- ---- ---------------
- all 27 0.81 0.62 1 of 5
- de 19 0.79 0.58 -
- en 8 0.88 0.71 -
+ RETRIEVAL METRICS (golden set: 32 questions, 5 unanswerable, 0 retrieval errors)
+ Subset n doc hit@5 passage hit@5 MRR false positives
+ ---------- --- --------- ------------- ---- ---------------
+ all 27 0.81 0.70 0.55 1 of 5
+ de 19 0.79 0.68 0.52 -
+ en 8 0.88 0.75 0.63 -
Faithfulness (10 answers, <model id>): 7 faithful, 2 partly, 1 wrong
- COST (prices read 2026-08-14)
- Item Monthly Per 1,000 queries
- ---------------------------- ---------- -----------------
- Vector store (2 OCU + 6 GB) $351.00 $39.00
- Full re-embed (one-off) $18.40 -
- Query embeddings $0.20 $0.02
- Generation (<model id>) $61.00 $6.80
- Total at 9,000 queries/month $412.20 $45.82
+ COST (prices read 2026-08-14; store cost from Cost Explorer, tag kb=docs-prod)
+ Item Monthly Per 1,000 queries
+ ------------------------------ ---------- -----------------
+ Vector store (min 2 OCU + 6 GB) $351.00 $39.00
+ Semantic chunking not configured
+ Parsing default parser, no charge
+ Full re-embed (one-off) $18.40 -
+ Query embeddings $0.20 $0.02
+ Reranking not configured
+ Store requests included in OCU capacity above
+ Generation (<model id>) $61.00 $6.80
+ Total at 9,000 queries/month $412.20 $45.82
Eval set: ~/.vibestack/projects/<slug>/kb-eval-2026-08-14.jsonl
Results: ~/.vibestack/projects/<slug>/kb-eval-results-2026-08-14.jsonl
Report: ~/.vibestack/projects/<slug>/kb-review-2026-08-14.md
```
Severity labels: `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFO`. Every finding names a
file, a resource id, or a command output as evidence.
---
## Important Rules
1. **Read-only against AWS.** This skill calls get, list, describe, retrieve and retrieve-and-generate. It never starts an ingestion job, never creates, updates or deletes a knowledge base or data source, never writes to S3, and never changes a vector index. The fixes in the findings are for the user to apply.
2. **Never start a sync.** Even when the last job is stale. Report it; a sync costs embedding tokens and can surface documents the user did not mean to publish yet.
3. **Sample, do not dump.** Copy at most five documents per format into the scratchpad, read them there, and quote at most one sentence per finding. Document contents never go into the report or into learnings.
4. **The eval set is the deliverable.** Write it before scoring anything, keep it in the project directory under `~/.vibestack`, and print its path. The next review reuses it so the numbers are comparable.
5. **Say what the runtime calls cost.** Retrieve and generate calls are billed. State the count before running the golden set and stay within it.
- 6. **Isolation findings are never downgraded.** A missing tenant filter is CRITICAL regardless of how few tenants exist today.
+ 6. **Isolation findings are never downgraded.** A missing tenant filter, or a user context the caller can set, is CRITICAL regardless of how few tenants exist today. A retrieve that exits non-zero proves nothing either way: report it N-A, never as a pass.
7. **Prices carry a date.** Unit prices change; every cost figure names the day the price was read and the volume it assumes.
8. **No account ids or document text in learnings.** Refer to the account by alias or slug and to documents by their role, never by content.
{{include lib/snippets/capture-learnings.md}}