llms.txt@11-RAG-Search · git:20260929.68bc267 · 2026-09-29 · sha256 4dd136f59ca89fa8
llms.txt@11-RAG-Search git:20260929.68bc267B
Immutable. This exact content never changes and is served at /api/v1/blob/4dd136f59ca89fa8.
# RAG & Search
> Every page below, concatenated. Each section gives the page title, its upstream source and its path in this repository.
---
## Enhancing RAG with Contextual Retrieval
Source: https://github.com/anthropics/claude-cookbooks/blob/main/capabilities/contextual-embeddings/guide.ipynb
Path: 11-RAG-Search/claude-cookbooks-capabilities-contextual-embeddings-guide.md
# Enhancing RAG with Contextual Retrieval
> Note: For more background information on Contextual Retrieval,
> including additional performance evaluations on various datasets, we
> recommend reading our accompanying [blog
> post](https://www.anthropic.com/news/contextual-retrieval).
Retrieval Augmented Generation (RAG) enables Claude to leverage your
internal knowledge bases, codebases, or any other corpus of documents
when providing a response. Enterprises are increasingly building RAG
applications to improve workflows in customer support, Q&A over internal
company documents, financial & legal analysis, code generation, and much
more.
In a [separate
guide](https://github.com/anthropics/anthropic-cookbook/blob/main/capabilities/retrieval_augmented_generation/guide.ipynb),
we walked through setting up a basic retrieval system, demonstrated how
to evaluate its performance, and then outlined a few techniques to
improve performance. In this guide, we present a technique for improving
retrieval performance: Contextual Embeddings.
In traditional RAG, documents are typically split into smaller chunks
for efficient retrieval. While this approach works well for many
applications, it can lead to problems when individual chunks lack
sufficient context. Contextual Embeddings solve this problem by adding
relevant context to each chunk before embedding. This method improves
the quality of each embedded chunk, allowing for more accurate retrieval
and thus better overall performance. Averaged across all data sources we
tested, Contextual Embeddings reduced the top-20-chunk retrieval failure
rate by 35%.
The same chunk-specific context can also be used with BM25 search to
further improve retrieval performance. We introduce this technique in
the \"Contextual BM25\" section.
In this guide, we\'ll demonstrate how to build and optimize a Contextual
Retrieval system using a dataset of 9 codebases as our knowledge base.
We\'ll walk through:
1\) Setting up a basic retrieval pipeline to establish a baseline for
performance.
2\) Contextual Embeddings: what it is, why it works, and how prompt
caching makes it practical for production use cases.
3\) Implementing Contextual Embeddings and demonstrating performance
improvements.
4\) Contextual BM25: improving performance with *contextual* BM25 hybrid
search.
5\) Improving performance with reranking,
### Evaluation Metrics & Dataset: {#evaluation-metrics--dataset}
We use a pre-chunked dataset of 9 codebases - all of which have been
chunked according to a basic character splitting mechanism. Our
evaluation dataset contains 248 queries - each of which contains a
\'golden chunk.\' We\'ll use a metric called <Pass@k> to evaluate
performance. <Pass@k> checks whether or not the \'golden document\' was
present in the first k documents retrieved for each query. Contextual
Embeddings in this case helped us to improve <Pass@10> performance from
\~87% \--\> \~95%.
You can find the code files and their chunks in
`data/codebase_chunks.json` and the evaluation dataset in
`data/evaluation_set.jsonl`
#### Additional Notes:
Prompt caching is helpful in managing costs when using this retrieval
method. This feature is currently available on Anthropic\'s first-party
API, and is coming soon to our third-party partner environments in AWS
Bedrock and GCP Vertex. We know that many of our customers leverage AWS
Knowledge Bases and GCP Vertex AI APIs when building RAG solutions, and
this method can be used on either platform with a bit of customization.
Consider reaching out to Anthropic or your AWS/GCP account team for
guidance on this!
To make it easier to use this method on Bedrock, the AWS team has
provided us with code that you can use to implement a Lambda function
that adds context to each document. If you deploy this Lambda function,
you can select it as a custom chunking option when configuring a
[Bedrock Knowledge
Base](https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-create.html).
You can find this code in `contextual-rag-lambda-function`. The main
lambda function code is in `lambda_function.py`.
## Table of Contents
1\) Setup
2\) Basic RAG
3\) Contextual Embeddings
4\) Contextual BM25
5\) Reranking
:::
::: {.cell .markdown}
## Setup
Before starting this guide, ensure you have:
**Technical Skills:**
- Intermediate Python programming
- Basic understanding of RAG (Retrieval Augmented Generation)
- Familiarity with vector databases and embeddings
- Basic command-line proficiency
**System Requirements:**
- Python 3.8+
- Docker installed and running (optional, for BM25 search)
- 4GB+ available RAM
- \~5-10 GB disk space for vector databases
**API Access:**
- [Anthropic API key](https://console.anthropic.com/) (free tier
sufficient)
- [Voyage AI API key](https://www.voyageai.com/)
- [Cohere API key](https://cohere.com/)
**Time & Cost:**
- Expected completion time: 30-45 minutes
- API costs: \~\$5-10 to run through the full dataset
### Libraries
We\'ll need a few libraries, including:
1\) `anthropic` - to interact with Claude
2\) `voyageai` - to generate high quality embeddings
3\) `cohere` - for reranking
4\) `elasticsearch` for performant BM25 search
3\) `pandas`, `numpy`, `matplotlib`, and `scikit-learn` for data
manipulation and visualization
### Environment Variables
Ensure the following environment variables are set:
- VOYAGE_API_KEY
- ANTHROPIC_API_KEY
- COHERE_API_KEY
:::
::: {.cell .code execution_count="6"}
``` python
%%capture
!pip install --upgrade anthropic voyageai cohere elasticsearch pandas numpy
```
:::
::: {.cell .markdown}
We define our model names up front to make it easier to change models as
new models are released
:::
::: {.cell .code}
``` python
MODEL_NAME = "claude-haiku-4-5"
```
:::
::: {.cell .markdown}
We\'ll start by initializing the Anthropic client that we\'ll use for
generating contextual descriptions.
:::
::: {.cell .code execution_count="5"}
``` python
import os
import anthropic
client = anthropic.Anthropic(
# This is the default and can be omitted
api_key=os.getenv("ANTHROPIC_API_KEY"),
)
```
:::
::: {.cell .markdown}
## Initialize a Vector DB Class
We\'ll create a VectorDB class to handle embedding storage and
similarity search. This class serves three key functions in our RAG
pipeline:
1. **Embedding Generation**: Converts text chunks into vector
representations using Voyage AI\'s embedding model
2. **Storage & Caching**: Saves embeddings to disk to avoid
re-computing them (which saves time and API costs)
3. **Similarity Search**: Retrieves the most relevant chunks for a
given query using cosine similarity
For this guide, we\'re using a simple in-memory vector database with
pickle serialization. This makes the code easy to understand and
requires no external dependencies. The class automatically saves
embeddings to disk after generation, so you only pay the embedding cost
once.
For production use, consider hosted vector database solutions.
The VectorDB class below follows the same interface patterns you\'d use
with production solutions, making it easy to swap out later. Key
features include batch processing (128 chunks at a time), progress
tracking with tqdm, and query caching to speed up repeated searches
during evaluation.
:::
::: {.cell .code}
``` python
import json
import pickle
from typing import Any
import numpy as np
import voyageai
from tqdm import tqdm
class VectorDB:
def __init__(self, name: str, api_key=None):
if api_key is None:
api_key = os.getenv("VOYAGE_API_KEY")
self.client = voyageai.Client(api_key=api_key)
self.name = name
self.embeddings = []
self.metadata = []
self.query_cache = {}
self.db_path = f"./data/{name}/vector_db.pkl"
def load_data(self, dataset: list[dict[str, Any]]):
if self.embeddings and self.metadata:
print("Vector database is already loaded. Skipping data loading.")
return
if os.path.exists(self.db_path):
print("Loading vector database from disk.")
self.load_db()
return
texts_to_embed = []
metadata = []
total_chunks = sum(len(doc["chunks"]) for doc in dataset)
with tqdm(total=total_chunks, desc="Processing chunks") as pbar:
for doc in dataset:
for chunk in doc["chunks"]:
texts_to_embed.append(chunk["content"])
metadata.append(
{
"doc_id": doc["doc_id"],
"original_uuid": doc["original_uuid"],
"chunk_id": chunk["chunk_id"],
"original_index": chunk["original_index"],
"content": chunk["content"],
}
)
pbar.update(1)
self._embed_and_store(texts_to_embed, metadata)
self.save_db()
print(f"Vector database loaded and saved. Total chunks processed: {len(texts_to_embed)}")
def _embed_and_store(self, texts: list[str], data: list[dict[str, Any]]):
batch_size = 128
with tqdm(total=len(texts), desc="Embedding chunks") as pbar:
result = []
for i in range(0, len(texts), batch_size):
batch = texts[i : i + batch_size]
batch_result = self.client.embed(batch, model="voyage-2").embeddings
result.extend(batch_result)
pbar.update(len(batch))
self.embeddings = result
self.metadata = data
def search(self, query: str, k: int = 20) -> list[dict[str, Any]]:
if query in self.query_cache:
query_embedding = self.query_cache[query]
else:
query_embedding = self.client.embed([query], model="voyage-2").embeddings[0]
self.query_cache[query] = query_embedding
if not self.embeddings:
raise ValueError("No data loaded in the vector database.")
similarities = np.dot(self.embeddings, query_embedding)
top_indices = np.argsort(similarities)[::-1][:k]
top_results = []
for idx in top_indices:
result = {
"metadata": self.metadata[idx],
"similarity": float(similarities[idx]),
}
top_results.append(result)
return top_results
def save_db(self):
data = {
"embeddings": self.embeddings,
"metadata": self.metadata,
"query_cache": json.dumps(self.query_cache),
}
os.makedirs(os.path.dirname(self.db_path), exist_ok=True)
with open(self.db_path, "wb") as file:
pickle.dump(data, file)
def load_db(self):
if not os.path.exists(self.db_path):
raise ValueError(
"Vector database file not found. Use load_data to create a new database."
)
with open(self.db_path, "rb") as file:
data = pickle.load(file)
self.embeddings = data["embeddings"]
self.metadata = data["metadata"]
self.query_cache = json.loads(data["query_cache"])
```
:::
::: {.cell .markdown}
Now we can use this class to load our dataset
:::
:::::: {.cell .code execution_count="12"}
``` python
# Load your transformed dataset
with open("data/codebase_chunks.json") as f:
transformed_dataset = json.load(f)
# Initialize the VectorDB
base_db = VectorDB("base_db")
# Load and process the data
base_db.load_data(transformed_dataset)
```
::: {.output .stream .stderr}
Processing chunks: 100%|██████████| 737/737 [00:00<00:00, 985400.72it/s]
Embedding chunks: 100%|██████████| 737/737 [00:42<00:00, 17.28it/s]
:::
::: {.output .stream .stdout}
Vector database loaded and saved. Total chunks processed: 737
:::
::: {.output .stream .stderr}
:::
::::::
::: {.cell .markdown}
## Basic RAG
To get started, we\'ll set up a basic RAG pipeline using a bare bones
approach. This is sometimes called \'Naive RAG\' by many in the
industry. A basic RAG pipeline includes the following 3 steps:
1\) Chunk documents by heading - containing only the content from each
subheading
2\) Embed each document
3\) Use Cosine similarity to retrieve documents in order to answer query
:::
::: {.cell .code execution_count="26"}
``` python
import json
from collections.abc import Callable
from typing import Any
from tqdm import tqdm
def load_jsonl(file_path: str) -> list[dict[str, Any]]:
"""Load JSONL file and return a list of dictionaries."""
with open(file_path) as file:
return [json.loads(line) for line in file]
def evaluate_retrieval(
queries: list[dict[str, Any]], retrieval_function: Callable, db, k: int = 20
) -> dict[str, float]:
total_score = 0
total_queries = len(queries)
for query_item in tqdm(queries, desc="Evaluating retrieval"):
query = query_item["query"]
golden_chunk_uuids = query_item["golden_chunk_uuids"]
# Find all golden chunk contents
golden_contents = []
for doc_uuid, chunk_index in golden_chunk_uuids:
golden_doc = next(
(doc for doc in query_item["golden_documents"] if doc["uuid"] == doc_uuid), None
)
if not golden_doc:
print(f"Warning: Golden document not found for UUID {doc_uuid}")
continue
golden_chunk = next(
(chunk for chunk in golden_doc["chunks"] if chunk["index"] == chunk_index), None
)
if not golden_chunk:
print(
f"Warning: Golden chunk not found for index {chunk_index} in document {doc_uuid}"
)
continue
golden_contents.append(golden_chunk["content"].strip())
if not golden_contents:
print(f"Warning: No golden contents found for query: {query}")
continue
retrieved_docs = retrieval_function(query, db, k=k)
# Count how many golden chunks are in the top k retrieved documents
chunks_found = 0
for golden_content in golden_contents:
for doc in retrieved_docs[:k]:
retrieved_content = (
doc["metadata"]
.get("original_content", doc["metadata"].get("content", ""))
.strip()
)
if retrieved_content == golden_content:
chunks_found += 1
break
query_score = chunks_found / len(golden_contents)
total_score += query_score
average_score = total_score / total_queries
pass_at_n = average_score * 100
return {"pass_at_n": pass_at_n, "average_score": average_score, "total_queries": total_queries}
def retrieve_base(query: str, db, k: int = 20) -> list[dict[str, Any]]:
"""
Retrieve relevant documents using either VectorDB or ContextualVectorDB.
:param query: The query string
:param db: The VectorDB or ContextualVectorDB instance
:param k: Number of top results to retrieve
:return: List of retrieved documents
"""
return db.search(query, k=k)
def evaluate_db(db, original_jsonl_path: str, k):
# Load the original JSONL data for queries and ground truth
original_data = load_jsonl(original_jsonl_path)
# Evaluate retrieval
results = evaluate_retrieval(original_data, retrieve_base, db, k)
return results
def evaluate_and_display(db, jsonl_path: str, k_values: list[int] = None, db_name: str = ""):
"""
Evaluate retrieval performance across multiple k values and display formatted results.
Args:
db: Vector database instance (VectorDB or ContextualVectorDB)
jsonl_path: Path to evaluation dataset
k_values: List of k values to evaluate (default: [5, 10, 20])
db_name: Optional name for the database being evaluated
Returns:
Dict mapping k values to their results
"""
if k_values is None:
k_values = [5, 10, 20]
results = {}
print(f"{'=' * 60}")
if db_name:
print(f"Evaluation Results: {db_name}")
else:
print("Evaluation Results")
print(f"{'=' * 60}\n")
for k in k_values:
print(f"Evaluating Pass@{k}...")
results[k] = evaluate_db(db, jsonl_path, k)
print() # Add spacing between evaluations
# Print summary table
print(f"{'=' * 60}")
print(f"{'Metric':<15} {'Pass Rate':<15} {'Score':<15}")
print(f"{'-' * 60}")
for k in k_values:
pass_rate = f"{results[k]['pass_at_n']:.2f}%"
score = f"{results[k]['average_score']:.4f}"
print(f"{'Pass@' + str(k):<15} {pass_rate:<15} {score:<15}")
print(f"{'=' * 60}\n")
return results
```
:::
::: {.cell .markdown}
Now let\'s establish our baseline performance by evaluating the basic
RAG system. We\'ll test at k=5, 10, and 20 to see how many of the golden
chunks appear in the top retrieved results. This gives us a benchmark to
measure improvement against.
:::
::::::::::: {.cell .code}
``` python
results = evaluate_and_display(
base_db, "data/evaluation_set.jsonl", k_values=[5, 10, 20], db_name="Baseline RAG"
)
```
::: {.output .stream .stdout}
============================================================
Evaluation Results: Contextual Embeddings
============================================================
Evaluating Pass@5...
:::
::: {.output .stream .stderr}
Evaluating retrieval: 100%|██████████| 248/248 [00:03<00:00, 65.26it/s]
:::
::: {.output .stream .stdout}
Evaluating Pass@10...
:::
::: {.output .stream .stderr}
Evaluating retrieval: 100%|██████████| 248/248 [00:03<00:00, 64.87it/s]
:::
::: {.output .stream .stdout}
Evaluating Pass@20...
:::
::: {.output .stream .stderr}
Evaluating retrieval: 100%|██████████| 248/248 [00:03<00:00, 64.72it/s]
:::
::: {.output .stream .stdout}
============================================================
Metric Pass Rate Score
------------------------------------------------------------
Pass@5 80.92% 0.8092
Pass@10 87.15% 0.8715
Pass@20 90.06% 0.9006
============================================================
:::
::: {.output .stream .stderr}
:::
:::::::::::
::: {.cell .markdown}
These results show our baseline RAG performance. The system successfully
retrieves the correct chunk 81% of the time in the top 5 results,
improving to 87% in the top 10, and 90% in the top 20.
:::
::: {.cell .markdown}
## Contextual Embeddings
With basic RAG, individual chunks often lack sufficient context when
embedded in isolation. Contextual Embeddings solve this by using Claude
to generate a brief description that \"situates\" each chunk within its
source document. We then embed the chunk together with this context,
creating richer vector representations.
For each chunk in our codebase dataset, we pass both the chunk and its
full source file to Claude. Claude generates a concise explanation of
what the chunk contains and where it fits in the overall file. This
context gets prepended to the chunk before embedding.
### Cost and Latency Considerations
**When does this cost occur?** The contextualization happens once at
ingestion time, not during every query. Unlike techniques like HyDE
(hypothetical document embeddings) that add latency to each search,
contextual embeddings are a one-time cost when building your vector
database. Prompt caching makes this practical. Since we process all
chunks from the same document sequentially, we can leverage prompt
caching for significant savings.
1. First chunk: We write the full document to cache (pay a small
premium)
2. Subsequent chunks: Read the document from cache (90% discount on
those tokens)
3. Cache lasts 5 minutes, plenty of time to process all chunks in a
document
**Cost example**: For 800-token chunks in 8k-token documents with 100
tokens of generated context, the total cost is \$1.02 per million
document tokens. You\'ll see the cache savings in the logs when you run
the code below.
**Note:** Some embedding models have fixed input token limits. If you
see worse performance with contextual embeddings, your contextualized
chunks may be getting truncated---consider using an embedding model with
a larger context window.
:::
::: {.cell .markdown}
------------------------------------------------------------------------
Let\'s see an example of how contextual embeddings work by generating
context for a single chunk. We\'ll use Claude to create a situating
context, and you\'ll also see the prompt caching metrics in action.
:::
:::: {.cell .code}
``` python
DOCUMENT_CONTEXT_PROMPT = """
<document>
{doc_content}
</document>
"""
CHUNK_CONTEXT_PROMPT = """
Here is the chunk we want to situate within the whole document
<chunk>
{chunk_content}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.
"""
def situate_context(doc: str, chunk: str) -> str:
response = client.messages.create(
model=MODEL_NAME,
max_tokens=1024,
temperature=0.0,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": DOCUMENT_CONTEXT_PROMPT.format(doc_content=doc),
"cache_control": {
"type": "ephemeral"
}, # we will make use of prompt caching for the full documents
},
{
"type": "text",
"text": CHUNK_CONTEXT_PROMPT.format(chunk_content=chunk),
},
],
}
],
)
return response
jsonl_data = load_jsonl("data/evaluation_set.jsonl")
# Example usage
doc_content = jsonl_data[0]["golden_documents"][0]["content"]
chunk_content = jsonl_data[0]["golden_chunks"][0]["content"]
response = situate_context(doc_content, chunk_content)
print(f"Situated context: {response.content[0].text}")
print("-" * 10)
# Print cache performance metrics
print(f"Input tokens: {response.usage.input_tokens}")
print(f"Output tokens: {response.usage.output_tokens}")
print(f"Cache creation input tokens: {response.usage.cache_creation_input_tokens}")
print(f"Cache read input tokens: {response.usage.cache_read_input_tokens}")
```
::: {.output .stream .stdout}
Situated context: This chunk contains the module documentation and initial struct definition for a differential fuzzing executor. It introduces the `DiffExecutor` struct that wraps two executors (primary and secondary) to run them sequentially with the same input, comparing their behavior for differential testing. The chunk establishes the core data structure and imports needed for the differential fuzzing implementation.
----------
Input tokens: 3412
Output tokens: 76
Cache creation input tokens: 0
Cache read input tokens: 0
:::
::::
::: {.cell .markdown}
### Building the Contextual Vector Database
Now that we\'ve seen how to generate contextual descriptions for
individual chunks, let\'s scale this up to process our entire dataset.
The `ContextualVectorDB` class below extends our basic `VectorDB` with
automatic contextualization during ingestion.
**Key features:**
- **Parallel processing**: Uses ThreadPoolExecutor to contextualize
multiple chunks simultaneously (configurable thread count)
- **Automatic prompt caching**: Processes chunks document-by-document to
maximize cache hits
- **Token tracking**: Monitors cache performance and calculates actual
cost savings
- **Persistent storage**: Saves both embeddings and contextualized
metadata to disk
When you run this, pay attention to the token usage statistics---you\'ll
see that 70-80% of input tokens are read from cache, demonstrating the
dramatic cost savings from prompt caching. On our 737-chunk dataset,
this reduces what would be a \~\$15 ingestion job down to \~\$3.
:::
::: {.cell .code execution_count="21"}
``` python
import json
import os
import pickle
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Any
import anthropic
import numpy as np
import voyageai
from tqdm import tqdm
class ContextualVectorDB:
def __init__(self, name: str, voyage_api_key=None, anthropic_api_key=None):
if voyage_api_key is None:
voyage_api_key = os.getenv("VOYAGE_API_KEY")
if anthropic_api_key is None:
anthropic_api_key = os.getenv("ANTHROPIC_API_KEY")
self.voyage_client = voyageai.Client(api_key=voyage_api_key)
self.anthropic_client = anthropic.Anthropic(api_key=anthropic_api_key)
self.name = name
self.embeddings = []
self.metadata = []
self.query_cache = {}
self.db_path = f"./data/{name}/contextual_vector_db.pkl"
self.token_counts = {"input": 0, "output": 0, "cache_read": 0, "cache_creation": 0}
self.token_lock = threading.Lock()
def situate_context(self, doc: str, chunk: str) -> tuple[str, Any]:
DOCUMENT_CONTEXT_PROMPT = """
<document>
{doc_content}
</document>
"""
CHUNK_CONTEXT_PROMPT = """
Here is the chunk we want to situate within the whole document
<chunk>
{chunk_content}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.
"""
response = self.anthropic_client.messages.create(
model=MODEL_NAME,
max_tokens=1000,
temperature=0.0,
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": DOCUMENT_CONTEXT_PROMPT.format(doc_content=doc),
"cache_control": {
"type": "ephemeral"
}, # we will make use of prompt caching for the full documents
},
{
"type": "text",
"text": CHUNK_CONTEXT_PROMPT.format(chunk_content=chunk),
},
],
},
],
extra_headers={"anthropic-beta": "prompt-caching-2024-07-31"},
)
return response.content[0].text, response.usage
def load_data(self, dataset: list[dict[str, Any]], parallel_threads: int = 1):
if self.embeddings and self.metadata:
print("Vector database is already loaded. Skipping data loading.")
return
if os.path.exists(self.db_path):
print("Loading vector database from disk.")
self.load_db()
return
texts_to_embed = []
metadata = []
total_chunks = sum(len(doc["chunks"]) for doc in dataset)
def process_chunk(doc, chunk):
# for each chunk, produce the context
contextualized_text, usage = self.situate_context(doc["content"], chunk["content"])
with self.token_lock:
self.token_counts["input"] += usage.input_tokens
self.token_counts["output"] += usage.output_tokens
self.token_counts["cache_read"] += usage.cache_read_input_tokens
self.token_counts["cache_creation"] += usage.cache_creation_input_tokens
return {
# append the context to the original text chunk
"text_to_embed": f"{chunk['content']}\n\n{contextualized_text}",
"metadata": {
"doc_id": doc["doc_id"],
"original_uuid": doc["original_uuid"],
"chunk_id": chunk["chunk_id"],
"original_index": chunk["original_index"],
"original_content": chunk["content"],
"contextualized_content": contextualized_text,
},
}
print(f"Processing {total_chunks} chunks with {parallel_threads} threads")
with ThreadPoolExecutor(max_workers=parallel_threads) as executor:
futures = []
for doc in dataset:
for chunk in doc["chunks"]:
futures.append(executor.submit(process_chunk, doc, chunk))
for future in tqdm(as_completed(futures), total=total_chunks, desc="Processing chunks"):
result = future.result()
texts_to_embed.append(result["text_to_embed"])
metadata.append(result["metadata"])
self._embed_and_store(texts_to_embed, metadata)
self.save_db()
# logging token usage
print(
f"Contextual Vector database loaded and saved. Total chunks processed: {len(texts_to_embed)}"
)
print(f"Total input tokens without caching: {self.token_counts['input']}")
print(f"Total output tokens: {self.token_counts['output']}")
print(f"Total input tokens written to cache: {self.token_counts['cache_creation']}")
print(f"Total input tokens read from cache: {self.token_counts['cache_read']}")
total_tokens = (
self.token_counts["input"]
+ self.token_counts["cache_read"]
+ self.token_counts["cache_creation"]
)
savings_percentage = (
(self.token_counts["cache_read"] / total_tokens) * 100 if total_tokens > 0 else 0
)
print(
f"Total input token savings from prompt caching: {savings_percentage:.2f}% of all input tokens used were read from cache."
)
print("Tokens read from cache come at a 90 percent discount!")
# we use voyage AI here for embeddings. Read more here: https://docs.voyageai.com/docs/embeddings
def _embed_and_store(self, texts: list[str], data: list[dict[str, Any]]):
batch_size = 128
result = [
self.voyage_client.embed(texts[i : i + batch_size], model="voyage-2").embeddings
for i in range(0, len(texts), batch_size)
]
self.embeddings = [embedding for batch in result for embedding in batch]
self.metadata = data
def search(self, query: str, k: int = 20) -> list[dict[str, Any]]:
if query in self.query_cache:
query_embedding = self.query_cache[query]
else:
query_embedding = self.voyage_client.embed([query], model="voyage-2").embeddings[0]
self.query_cache[query] = query_embedding
if not self.embeddings:
raise ValueError("No data loaded in the vector database.")
similarities = np.dot(self.embeddings, query_embedding)
top_indices = np.argsort(similarities)[::-1][:k]
top_results = []
for idx in top_indices:
result = {
"metadata": self.metadata[idx],
"similarity": float(similarities[idx]),
}
top_results.append(result)
return top_results
def save_db(self):
data = {
"embeddings": self.embeddings,
"metadata": self.metadata,
"query_cache": json.dumps(self.query_cache),
}
os.makedirs(os.path.dirname(self.db_path), exist_ok=True)
with open(self.db_path, "wb") as file:
pickle.dump(data, file)
def load_db(self):
if not os.path.exists(self.db_path):
raise ValueError(
"Vector database file not found. Use load_data to create a new database."
)
with open(self.db_path, "rb") as file:
data = pickle.load(file)
self.embeddings = data["embeddings"]
self.metadata = data["metadata"]
self.query_cache = json.loads(data["query_cache"])
```
:::
:::::: {.cell .code execution_count="22"}
``` python
# Load the transformed dataset
with open("data/codebase_chunks.json") as f:
transformed_dataset = json.load(f)
# Initialize the ContextualVectorDB
contextual_db = ContextualVectorDB("my_contextual_db")
# Load and process the data
# note: consider increasing the number of parallel threads to run this faster, or reducing the number of parallel threads if concerned about hitting your API rate limit
contextual_db.load_data(transformed_dataset, parallel_threads=5)
```
::: {.output .stream .stdout}
Processing 737 chunks with 5 threads
:::
::: {.output .stream .stderr}
Processing chunks: 100%|██████████| 737/737 [05:32<00:00, 2.22it/s]
:::
::: {.output .stream .stdout}
Contextual Vector database loaded and saved. Total chunks processed: 737
Total input tokens without caching: 1223730
Total output tokens: 58161
Total input tokens written to cache: 176079
Total input tokens read from cache: 2267069
Total input token savings from prompt caching: 61.83% of all input tokens used were read from cache.
Tokens read from cache come at a 90 percent discount!
:::
::::::
::: {.cell .markdown}
These numbers reveal the power of prompt caching for contextual
embeddings:
- We processed **737 chunks** across 9 codebase files
- **61.83% of input tokens** were read from cache (2.27M tokens at 90%
discount)
- Without caching, this would cost **\~\$9.20** in input tokens
- With caching, the actual cost drops to **\~\$2.85** (69% savings)
The cache hit rate depends on how many chunks each document contains.
Files with more chunks benefit more from caching since we write the full
document to cache once, then read it repeatedly for each chunk in that
file. This is why processing documents sequentially (rather than
randomly shuffling chunks) is crucial for maximizing cache efficiency.
Now let\'s evaluate how much this contextualization improves our
retrieval performance compared to the baseline.
:::
::::::::::: {.cell .code execution_count="28"}
``` python
results = evaluate_and_display(
contextual_db,
"data/evaluation_set.jsonl",
k_values=[5, 10, 20],
db_name="Contextual Embeddings",
)
```
::: {.output .stream .stdout}
============================================================
Evaluation Results: Contextual Embeddings
============================================================
Evaluating Pass@5...
:::
::: {.output .stream .stderr}
Evaluating retrieval: 100%|██████████| 248/248 [00:03<00:00, 64.58it/s]
:::
::: {.output .stream .stdout}
Evaluating Pass@10...
:::
::: {.output .stream .stderr}
Evaluating retrieval: 100%|██████████| 248/248 [00:03<00:00, 64.37it/s]
:::
::: {.output .stream .stdout}
Evaluating Pass@20...
:::
::: {.output .stream .stderr}
Evaluating retrieval: 100%|██████████| 248/248 [00:03<00:00, 64.14it/s]
:::
::: {.output .stream .stdout}
============================================================
Metric Pass Rate Score
------------------------------------------------------------
Pass@5 88.12% 0.8812
Pass@10 92.34% 0.9234
Pass@20 94.29% 0.9429
============================================================
:::
::: {.output .stream .stderr}
:::
:::::::::::
::: {.cell .markdown}
By adding context to each chunk before embedding, we\'ve reduced
retrieval failures by **\~30-40%** across all k values. This means fewer
irrelevant results in your top retrieved chunks, leading to better
answers when you pass these chunks to Claude for final response
generation.
The improvement is most pronounced at <Pass@5>, where precision matters
most---suggesting that contextualized chunks aren\'t just retrieved more
often, but rank higher when relevant.
:::
::: {.cell .markdown}
## Contextual BM25: Hybrid Search
Contextual embeddings alone improved our <Pass@10> from 87% to 92%. We
can push performance even higher by combining semantic search with
keyword-based search using **Contextual BM25**---a hybrid approach that
reduces retrieval failure rates further.
### Why Hybrid Search?
Semantic search excels at understanding meaning and context, but can
miss exact keyword matches. BM25 (a probabilistic keyword ranking
algorithm) excels at finding specific terms, but lacks semantic
understanding. By combining both, we get the best of both worlds:
- **Semantic search**: Captures conceptual similarity and paraphrases
- **BM25**: Catches exact terminology, function names, and specific
phrases
- **Reciprocal Rank Fusion**: Intelligently merges results from both
sources
### What is BM25?
BM25 is a probabilistic ranking function that improves upon TF-IDF by
accounting for document length and term saturation. It\'s widely used in
production search engines (including Elasticsearch) for its
effectiveness at ranking keyword relevance. For technical details, see
[this blog
post](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables).
Instead of only searching the raw chunk content, we search both the
chunk *and* the contextual description we generated earlier. This means
BM25 can match keywords in either the original text or the explanatory
context.
### Setup: Running Elasticsearch
Before running the code below, you\'ll need Elasticsearch running
locally. The easiest way is with Docker:
``` bash
docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" \
-e "xpack.security.enabled=false" \
elasticsearch:9.2.0
```
## Troubleshooting:
- Verify it\'s running: docker ps \| grep elasticsearch
- If port 9200 is in use: docker stop elasticsearch && docker rm
elasticsearch
- Check logs if issues occur: docker logs elasticsearch
:::
::: {.cell .markdown}
## How the Hybrid Search Works
The retrieve_advanced function below implements a three-step process:
1. Retrieve candidates: Get top 150 results from both semantic search
and BM25
2. Score fusion: Combine rankings using weighted Reciprocal Rank Fusion
- Default: 80% weight to semantic search, 20% to BM25
- These weights are tunable---experiment to optimize for your use
case
3. Return top-k: Select the highest-scoring results after fusion
The weighting system lets you balance between semantic understanding and
keyword precision based on your data characteristics.
:::
::: {.cell .code}
``` python
import json
import os
from typing import Any
from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk
from tqdm import tqdm
class ElasticsearchBM25:
def __init__(self, index_name: str = "contextual_bm25_index"):
self.es_client = Elasticsearch("http://localhost:9200")
self.index_name = index_name
self.create_index()
def create_index(self):
index_settings = {
"settings": {
"analysis": {"analyzer": {"default": {"type": "english"}}},
"similarity": {"default": {"type": "BM25"}},
"index.queries.cache.enabled": False,
},
"mappings": {
"properties": {
"content": {"type": "text", "analyzer": "english"},
"contextualized_content": {"type": "text", "analyzer": "english"},
"doc_id": {"type": "keyword", "index": False},
"chunk_id": {"type": "keyword", "index": False},
"original_index": {"type": "integer", "index": False},
}
},
}
# Change this line - remove 'body=' parameter
if not self.es_client.indices.exists(index=self.index_name):
self.es_client.indices.create(
index=self.index_name,
settings=index_settings["settings"],
mappings=index_settings["mappings"],
)
print(f"Created index: {self.index_name}")
def index_documents(self, documents: list[dict[str, Any]]):
actions = [
{
"_index": self.index_name,
"_source": {
"content": doc["original_content"],
"contextualized_content": doc["contextualized_content"],
"doc_id": doc["doc_id"],
"chunk_id": doc["chunk_id"],
"original_index": doc["original_index"],
},
}
for doc in documents
]
success, _ = bulk(self.es_client, actions)
self.es_client.indices.refresh(index=self.index_name)
return success
def search(self, query: str, k: int = 20) -> list[dict[str, Any]]:
self.es_client.indices.refresh(index=self.index_name)
# Change this - remove 'body=' and pass query directly
response = self.es_client.search(
index=self.index_name,
query={
"multi_match": {
"query": query,
"fields": ["content", "contextualized_content"],
}
},
size=k,
)
return [
{
"doc_id": hit["_source"]["doc_id"],
"original_index": hit["_source"]["original_index"],
"content": hit["_source"]["content"],
"contextualized_content": hit["_source"]["contextualized_content"],
"score": hit["_score"],
}
for hit in response["hits"]["hits"]
]
def create_elasticsearch_bm25_index(db: ContextualVectorDB):
es_bm25 = ElasticsearchBM25()
es_bm25.index_documents(db.metadata)
return es_bm25
def retrieve_advanced(
query: str,
db: ContextualVectorDB,
es_bm25: ElasticsearchBM25,
k: int,
semantic_weight: float = 0.8,
bm25_weight: float = 0.2,
):
num_chunks_to_recall = 150
# Semantic search
semantic_results = db.search(query, k=num_chunks_to_recall)
ranked_chunk_ids = [
(result["metadata"]["doc_id"], result["metadata"]["original_index"])
for result in semantic_results
]
# BM25 search using Elasticsearch
bm25_results = es_bm25.search(query, k=num_chunks_to_recall)
ranked_bm25_chunk_ids = [
(result["doc_id"], result["original_index"]) for result in bm25_results
]
# Combine results
chunk_ids = list(set(ranked_chunk_ids + ranked_bm25_chunk_ids))
chunk_id_to_score = {}
# Initial scoring with weights
for chunk_id in chunk_ids:
score = 0
if chunk_id in ranked_chunk_ids:
index = ranked_chunk_ids.index(chunk_id)
score += semantic_weight * (1 / (index + 1)) # Weighted 1/n scoring for semantic
if chunk_id in ranked_bm25_chunk_ids:
index = ranked_bm25_chunk_ids.index(chunk_id)
score += bm25_weight * (1 / (index + 1)) # Weighted 1/n scoring for BM25
chunk_id_to_score[chunk_id] = score
# Sort chunk IDs by their scores in descending order
sorted_chunk_ids = sorted(
chunk_id_to_score.keys(), key=lambda x: (chunk_id_to_score[x], x[0], x[1]), reverse=True
)
# Assign new scores based on the sorted order
for index, chunk_id in enumerate(sorted_chunk_ids):
chunk_id_to_score[chunk_id] = 1 / (index + 1)
# Prepare the final results
final_results = []
semantic_count = 0
bm25_count = 0
for chunk_id in sorted_chunk_ids[:k]:
chunk_metadata = next(
chunk
for chunk in db.metadata
if chunk["doc_id"] == chunk_id[0] and chunk["original_index"] == chunk_id[1]
)
is_from_semantic = chunk_id in ranked_chunk_ids
is_from_bm25 = chunk_id in ranked_bm25_chunk_ids
final_results.append(
{
"chunk": chunk_metadata,
"score": chunk_id_to_score[chunk_id],
"from_semantic": is_from_semantic,
"from_bm25": is_from_bm25,
}
)
if is_from_semantic and not is_from_bm25:
semantic_count += 1
elif is_from_bm25 and not is_from_semantic:
bm25_count += 1
else: # it's in both
semantic_count += 0.5
bm25_count += 0.5
return final_results, semantic_count, bm25_count
def evaluate_db_advanced(
db: ContextualVectorDB,
original_jsonl_path: str,
k_values: list[int] = None,
db_name: str = "Hybrid Search",
):
"""
Evaluate hybrid search (semantic + BM25) at multiple k values with formatted results.
Args:
db: ContextualVectorDB instance
original_jsonl_path: Path to evaluation dataset
k_values: List of k values to evaluate (default: [5, 10, 20])
db_name: Name for the evaluation display
Returns:
Dict mapping k values to their results and source breakdowns
"""
if k_values is None:
k_values = [5, 10, 20]
original_data = load_jsonl(original_jsonl_path)
es_bm25 = create_elasticsearch_bm25_index(db)
results = {}
print(f"{'=' * 70}")
print(f"Evaluation Results: {db_name}")
print(f"{'=' * 70}\n")
try:
# Warm-up queries
warm_up_queries = original_data[:10]
for query_item in warm_up_queries:
_ = retrieve_advanced(query_item["query"], db, es_bm25, k_values[0])
for k in k_values:
print(f"Evaluating Pass@{k}...")
total_score = 0
total_semantic_count = 0
total_bm25_count = 0
total_results = 0
for query_item in tqdm(original_data, desc=f"Pass@{k}"):
query = query_item["query"]
golden_chunk_uuids = query_item["golden_chunk_uuids"]
golden_contents = []
for doc_uuid, chunk_index in golden_chunk_uuids:
golden_doc = next(
(doc for doc in query_item["golden_documents"] if doc["uuid"] == doc_uuid),
None,
)
if golden_doc:
golden_chunk = next(
(
chunk
for chunk in golden_doc["chunks"]
if chunk["index"] == chunk_index
),
None,
)
if golden_chunk:
golden_contents.append(golden_chunk["content"].strip())
if not golden_contents:
continue
retrieved_docs, semantic_count, bm25_count = retrieve_advanced(
query, db, es_bm25, k
)
chunks_found = 0
for golden_content in golden_contents:
for doc in retrieved_docs[:k]:
retrieved_content = doc["chunk"]["original_content"].strip()
if retrieved_content == golden_content:
chunks_found += 1
break
query_score = chunks_found / len(golden_contents)
total_score += query_score
total_semantic_count += semantic_count
total_bm25_count += bm25_count
total_results += len(retrieved_docs)
total_queries = len(original_data)
average_score = total_score / total_queries
pass_at_n = average_score * 100
semantic_percentage = (
(total_semantic_count / total_results) * 100 if total_results > 0 else 0
)
bm25_percentage = (total_bm25_count / total_results) * 100 if total_results > 0 else 0
results[k] = {
"pass_at_n": pass_at_n,
"average_score": average_score,
"total_queries": total_queries,
"semantic_percentage": semantic_percentage,
"bm25_percentage": bm25_percentage,
}
print(f"Pass@{k}: {pass_at_n:.2f}%")
print(f"Semantic: {semantic_percentage:.1f}% | BM25: {bm25_percentage:.1f}%\n")
# Print summary table
print(f"{'=' * 70}")
print(f"{'Metric':<12} {'Pass Rate':<12} {'Score':<12} {'Semantic':<12} {'BM25':<12}")
print(f"{'-' * 70}")
for k in k_values:
r = results[k]
print(
f"{'Pass@' + str(k):<12} {r['pass_at_n']:>10.2f}% {r['average_score']:>10.4f} "
f"{r['semantic_percentage']:>10.1f}% {r['bm25_percentage']:>10.1f}%"
)
print(f"{'=' * 70}\n")
return results
finally:
# Delete the Elasticsearch index
if es_bm25.es_client.indices.exists(index=es_bm25.index_name):
es_bm25.es_client.indices.delete(index=es_bm25.index_name)
print(f"Deleted Elasticsearch index: {es_bm25.index_name}")
```
:::
:::::::::: {.cell .code execution_count="39"}
``` python
results = evaluate_db_advanced(
contextual_db,
"data/evaluation_set.jsonl",
k_values=[5, 10, 20],
db_name="Contextual BM25 Hybrid Search",
)
```
::: {.output .stream .stdout}
Created index: contextual_bm25_index
======================================================================
Evaluation Results: Contextual BM25 Hybrid Search
======================================================================
Evaluating Pass@5...
:::
::: {.output .stream .stderr}
Pass@5: 100%|██████████| 248/248 [00:05<00:00, 41.79it/s]
:::
::: {.output .stream .stdout}
Pass@5: 88.86%
Semantic: 54.6% | BM25: 45.4%
Evaluating Pass@10...
:::
::: {.output .stream .stderr}
Pass@10: 100%|██████████| 248/248 [00:05<00:00, 42.20it/s]
:::
::: {.output .stream .stdout}
Pass@10: 92.31%
Semantic: 57.6% | BM25: 42.4%
Evaluating Pass@20...
:::
::: {.output .stream .stderr}
Pass@20: 100%|██████████| 248/248 [00:05<00:00, 42.15it/s]
:::
::: {.output .stream .stdout}
Pass@20: 95.23%
Semantic: 60.8% | BM25: 39.2%
======================================================================
Metric Pass Rate Score Semantic BM25
----------------------------------------------------------------------
Pass@5 88.86% 0.8886 54.6% 45.4%
Pass@10 92.31% 0.9231 57.6% 42.4%
Pass@20 95.23% 0.9523 60.8% 39.2%
======================================================================
Deleted Elasticsearch index: contextual_bm25_index
:::
::::::::::
::: {.cell .markdown}
## Reranking
We\'ve achieved strong results with hybrid search (93.21% <Pass@10>),
but there\'s one more technique that can squeeze out additional
performance: **reranking**.
### What is Reranking?
Reranking is a two-stage retrieval approach:
1. **Stage 1 - Broad Retrieval**: Cast a wide net by retrieving more
candidates than you need (e.g., retrieve 100 chunks)
2. **Stage 2 - Precise Selection**: Use a specialized reranking model
to score these candidates and select only the top-k most relevant
ones
**Why does this work?** Initial retrieval methods (embeddings, BM25) are
optimized for speed across millions of documents. Reranking models are
slower but more accurate---they can afford to do deeper analysis on a
smaller candidate set. This creates a speed/accuracy trade-off that
works well in practice.
### Our Reranking Approach
For this example, we\'ll use a simpler reranking pipeline that builds on
contextual embeddings alone (not the full hybrid search). Here\'s the
process:
1. **Over-retrieve**: Get 10x more results than needed (e.g., retrieve
100 chunks when we need 10)
2. **Rerank with Cohere**: Use Cohere\'s `rerank-english-v3.0` model to
score all candidates
3. **Select top-k**: Return only the highest-scoring results
The reranking model has access to both the original chunk content and
the contextual descriptions we generated, giving it rich information to
make precise relevance judgments.
### Expected Performance
Adding reranking delivers a modest but meaningful improvement:
- **Without reranking**: 92.34% <Pass@10> (contextual embeddings alone)
- **With reranking**: \~95% <Pass@10> (additional 2-3% gain)
This might seem small, but in production systems, reducing failures from
7.66% to \~5% can significantly improve user experience. The trade-off
is query latency---reranking adds \~100-200ms per query depending on
candidate set size.
:::
::: {.cell .code}
``` python
import json
from collections.abc import Callable
from typing import Any
import cohere
from tqdm import tqdm
def evaluate_db_rerank(
db, original_jsonl_path: str, k_values: list[int] = None, db_name: str = "Reranking"
):
"""
Evaluate reranking performance at multiple k values with formatted results.
Args:
db: ContextualVectorDB instance
original_jsonl_path: Path to evaluation dataset
k_values: List of k values to evaluate (default: [5, 10, 20])
db_name: Name for the evaluation display
Returns:
Dict mapping k values to their results
"""
if k_values is None:
k_values = [5, 10, 20]
original_data = load_jsonl(original_jsonl_path)
co = cohere.Client(os.getenv("COHERE_API_KEY"))
results = {}
print(f"{'=' * 60}")
print(f"Evaluation Results: {db_name}")
print(f"{'=' * 60}\n")
for k in k_values:
print(f"Evaluating Pass@{k} with reranking...")
total_score = 0
total_queries = len(original_data)
for query_item in tqdm(original_data, desc=f"Pass@{k}"):
query = query_item["query"]
golden_chunk_uuids = query_item["golden_chunk_uuids"]
# Find golden contents
golden_contents = []
for doc_uuid, chunk_index in golden_chunk_uuids:
golden_doc = next(
(doc for doc in query_item["golden_documents"] if doc["uuid"] == doc_uuid), None
)
if golden_doc:
golden_chunk = next(
(chunk for chunk in golden_doc["chunks"] if chunk["index"] == chunk_index),
None,
)
if golden_chunk:
golden_contents.append(golden_chunk["content"].strip())
if not golden_contents:
continue
# Retrieve and rerank
semantic_results = db.search(query, k=k * 10)
# Prepare documents for reranking
documents = [
f"{res['metadata']['original_content']}\n\nContext: {res['metadata']['contextualized_content']}"
for res in semantic_results
]
# Rerank
rerank_response = co.rerank(
model="rerank-english-v3.0", query=query, documents=documents, top_n=k
)
time.sleep(0.1) # Rate limiting
# Get final results
retrieved_docs = []
for r in rerank_response.results:
original_result = semantic_results[r.index]
retrieved_docs.append(
{"chunk": original_result["metadata"], "score": r.relevance_score}
)
# Check if golden chunks are in results
chunks_found = 0
for golden_content in golden_contents:
for doc in retrieved_docs[:k]:
retrieved_content = doc["chunk"]["original_content"].strip()
if retrieved_content == golden_content:
chunks_found += 1
break
query_score = chunks_found / len(golden_contents)
total_score += query_score
average_score = total_score / total_queries
pass_at_n = average_score * 100
results[k] = {
"pass_at_n": pass_at_n,
"average_score": average_score,
"total_queries": total_queries,
}
print(f"Pass@{k}: {pass_at_n:.2f}%")
print(f"Average Score: {average_score:.4f}\n")
# Print summary table
print(f"{'=' * 60}")
print(f"{'Metric':<15} {'Pass Rate':<15} {'Score':<15}")
print(f"{'-' * 60}")
for k in k_values:
pass_rate = f"{results[k]['pass_at_n']:.2f}%"
score = f"{results[k]['average_score']:.4f}"
print(f"{'Pass@' + str(k):<15} {pass_rate:<15} {score:<15}")
print(f"{'=' * 60}\n")
return results
```
:::
::::::::::: {.cell .code execution_count="48"}
``` python
results = evaluate_db_rerank(
contextual_db,
"data/evaluation_set.jsonl",
k_values=[5, 10, 20],
db_name="Contextual Embeddings + Reranking",
)
```
::: {.output .stream .stdout}
============================================================
Evaluation Results: Contextual Embeddings + Reranking
============================================================
Evaluating Pass@5 with reranking...
:::
::: {.output .stream .stderr}
Pass@5: 100%|██████████| 248/248 [01:40<00:00, 2.47it/s]
:::
::: {.output .stream .stdout}
Pass@5: 92.15%
Average Score: 0.9215
Evaluating Pass@10 with reranking...
:::
::: {.output .stream .stderr}
Pass@10: 100%|██████████| 248/248 [02:29<00:00, 1.66it/s]
:::
::: {.output .stream .stdout}
Pass@10: 95.26%
Average Score: 0.9526
Evaluating Pass@20 with reranking...
:::
::: {.output .stream .stderr}
Pass@20: 100%|██████████| 248/248 [03:03<00:00, 1.35it/s]
:::
::: {.output .stream .stdout}
Pass@20: 97.45%
Average Score: 0.9745
============================================================
Metric Pass Rate Score
------------------------------------------------------------
Pass@5 92.15% 0.9215
Pass@10 95.26% 0.9526
Pass@20 97.45% 0.9745
============================================================
:::
::: {.output .stream .stderr}
:::
:::::::::::
::: {.cell .markdown}
Reranking delivers our strongest results, nearly eliminating retrieval
failures. Let\'s look at how each technique built upon the previous one
to achieve this improvement.
Starting from our baseline RAG system at 87% <Pass@10>, we\'ve climbed
to over 95% by systematically applying advanced retrieval techniques.
Each method addresses a different weakness: contextual embeddings solve
the \"isolated chunk\" problem, hybrid search catches keyword-specific
queries that embeddings miss, and reranking applies more sophisticated
relevance scoring to refine the final selection.
Approach <Pass@5> <Pass@10> <Pass@20>
----------------------------- ---------- ----------- -----------
**Baseline RAG** 80.92% 87.15% 90.06%
**+ Contextual Embeddings** 88.12% 92.34% 94.29%
**+ Hybrid Search (BM25)** 86.43% 93.21% 94.99%
**+ Reranking** 92.15% 95.26% 97.45%
**Key Takeaways:**
1. **Contextual embeddings provided the largest single improvement**
(+5-7 percentage points), validating that adding document-level
context to chunks significantly improves retrieval quality. This
technique alone gets you 90% of the way to optimal performance.
2. **Reranking achieves the highest absolute performance**, reaching
95.26% <Pass@10>---meaning the correct chunk appears in the top 10
results for 95% of queries. This represents a **47% reduction in
retrieval failures** compared to baseline RAG (from 12.85% failure
rate down to 4.74%).
3. **Trade-offs matter**: Each technique adds complexity and cost:
- Contextual embeddings: One-time ingestion cost (\~\$3 for this
dataset with prompt caching)
- Hybrid search: Requires Elasticsearch infrastructure and
maintenance
- Reranking: Adds 100-200ms query latency and per-query API costs
(\~\$0.002 per query)
4. **Choose your approach** based on your requirements:
- **High-volume, cost-sensitive**: Contextual embeddings alone (92%
<Pass@10>, no per-query costs)
- **Maximum accuracy, latency-tolerant**: Full reranking pipeline
(95% <Pass@10>, best precision)
- **Balanced production system**: Hybrid search for strong
performance without per-query costs (93% <Pass@10>)
For most production RAG systems, **contextual embeddings provide the
best performance-to-cost ratio**, delivering 92% <Pass@10> with only
one-time ingestion costs. Hybrid search and reranking are available when
you need that extra 2-3 percentage points of precision and can afford
the additional infrastructure or query costs.
### Next Steps and Key Takeaways
1\) We demonstrated how to use Contextual Embeddings to improve
retrieval performance, then delivered additional improvements with
Contextual BM25 and reranking.
2\) This example used codebases, but these methods also apply to other
data types such as internal company knowledge bases, financial & legal
content, educational content, and much more.
3\) If you are an AWS user, you can get started with the Lambda function
in `contextual-rag-lambda-function`, and if you\'re a GCP user you can
spin up your own Cloud Run instance and follow a similar pattern!
:::
---
## Citations
Source: https://github.com/anthropics/claude-cookbooks/blob/main/misc/using_citations.ipynb
Path: 11-RAG-Search/claude-cookbooks-misc-using-citations.md
# Citations
The Claude API features citation support that enables Claude to provide
detailed citations when answering questions about documents. Citations
are a valuable affordance in many LLM powered applications to help users
track and verify the sources of information in responses.
Citations are supported on:
- `claude-sonnet-4-5`
- `claude-3-5-haiku-20241022`
The citations feature is an alternative to prompt-based citation
techniques. Using this featue has the following advantages:
- Prompt-based techniques often require Claude to output full quotes
from the source document it intends to cite. This increases output
tokens and therefore cost.
- The citation feature will not return citations pointing to documents
or locations that were not provided as valid sources.
- While testing we found the citation feature to generate citations with
higher recall and percision than prompt based techniques.
The documentation for citations can be found
[here](https://docs.claude.com/en/docs/build-with-claude/citations).
:::
::: {.cell .markdown}
## Setup
First, let\'s install the required libraries and initalize our Anthropic
client.
:::
::: {.cell .code}
``` python
!pip install anthropic --quiet
```
:::
::: {.cell .code execution_count="56"}
``` python
import json
import os
import anthropic
ANTHROPIC_API_KEY = os.environ.get("ANTHROPIC_API_KEY")
# ANTHROPIC_API_KEY = "" # Put your API key here!
client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY)
```
:::
::: {.cell .markdown}
## Document Types
Citations support three different document types. The type of citation
outputted depends on the type of document being cited from:
- Plain text document citation → char location format
- PDF document citation → page location format
- Custom content document citation → content block location format
We will explore working with each of these in the examples below.
:::
::: {.cell .markdown}
### Plain Text Documents
With plain text document citations you provide your document as raw text
to the model. You can provide one or multiple documents. This text will
get automatically chunked into sentences. The model will cite these
sentences as appropriate. The model is able to cite multiple sentences
together at once in a single citation but will not cite text smaller
than a sentence.
Along with the outputted text the API response will include structured
data for all citations.
Let\'s see a complete example using a help center customer chatbot for a
made up company PetWorld.
:::
:::: {.cell .code execution_count="57"}
``` python
# Read all help center articles and create a list of documents
articles_dir = "./data/help_center_articles"
documents = []
for filename in sorted(os.listdir(articles_dir)):
if filename.endswith(".txt"):
with open(os.path.join(articles_dir, filename)) as f:
content = f.read()
# Split into title and body
title_line, body = content.split("\n", 1)
title = title_line.replace("title: ", "")
documents.append(
{
"type": "document",
"source": {"type": "text", "media_type": "text/plain", "data": body},
"title": title,
"citations": {"enabled": True},
}
)
QUESTION = "I just checked out, where is my order tracking number? Track package is not available on the website yet for my order."
# Add the question to the content
content = documents
response = client.messages.create(
model="claude-sonnet-4-5",
temperature=0.0,
max_tokens=1024,
system="You are a customer support bot working for PetWorld. Your task is to provide short, helpful answers to user questions. Since you are in a chat interface avoid providing extra details. You will be given access to PetWorld's help center articles to help you answer questions.",
messages=[
{"role": "user", "content": documents},
{
"role": "user",
"content": [{"type": "text", "text": f"Here is the user's question: {QUESTION}"}],
},
],
)
def visualize_raw_response(response):
raw_response = {"content": []}
print("\n" + "=" * 80 + "\nRaw response:\n" + "=" * 80)
for content in response.content:
if content.type == "text":
block = {"type": "text", "text": content.text}
if hasattr(content, "citations") and content.citations:
block["citations"] = []
for citation in content.citations:
citation_dict = {
"type": citation.type,
"cited_text": citation.cited_text,
"document_title": citation.document_title,
}
if citation.type == "page_location":
citation_dict.update(
{
"start_page_number": citation.start_page_number,
"end_page_number": citation.end_page_number,
}
)
block["citations"].append(citation_dict)
raw_response["content"].append(block)
return json.dumps(raw_response, indent=2)
print(visualize_raw_response(response))
```
::: {.output .stream .stdout}
================================================================================
Raw response:
================================================================================
{
"content": [
{
"type": "text",
"text": "Based on the documentation, I can explain why you don't see tracking yet: "
},
{
"type": "text",
"text": "You'll receive an email with your tracking number once your order ships. If you don't receive a tracking number within 48 hours of your order confirmation, please contact our customer support team for assistance.",
"citations": [
{
"type": "char_location",
"cited_text": "Once your order ships, you'll receive an email with a tracking number. ",
"document_title": "Order Tracking Information"
},
{
"type": "char_location",
"cited_text": "If you haven't received a tracking number within 48 hours of your order confirmation, please contact our customer support team.",
"document_title": "Order Tracking Information"
}
]
},
{
"type": "text",
"text": "\n\nSince you just checked out, your order likely hasn't shipped yet. Once it ships, you'll receive the tracking information via email."
}
]
}
:::
::::
::: {.cell .markdown}
#### Visualizing Citations
By leveraging the citation data, we can create UIs that:
1. Show users exactly where information comes from
2. Link directly to source documents
3. Highlight cited text in context
4. Build trust through transparent sourcing
Below is a simple visualization function that transforms Claude\'s
structured citations into a readable format with numbered references,
similar to academic papers.
The function takes Claude\'s response object and outputs:
- Text with numbered citation markers (e.g., \"The answer \[1\] includes
this fact \[2\]\")
- A numbered reference list showing each cited text and its source
document
:::
:::: {.cell .code execution_count="58"}
``` python
def visualize_citations(response):
"""
Takes a response object and returns a string with numbered citations.
Example output: "here is the plain text answer [1][2] here is some more text [3]"
with a list of citations below.
"""
# Dictionary to store unique citations
citations_dict = {}
citation_counter = 1
# Final formatted text
formatted_text = ""
citations_list = []
print("\n" + "=" * 80 + "\nFormatted response:\n" + "=" * 80)
for content in response.content:
if content.type == "text":
text = content.text
if hasattr(content, "citations") and content.citations:
# Sort citations by their appearance in the text
def get_sort_key(citation):
if hasattr(citation, "start_char_index"):
return citation.start_char_index
elif hasattr(citation, "start_page_number"):
return citation.start_page_number
elif hasattr(citation, "start_block_index"):
return citation.start_block_index
return 0 # fallback
sorted_citations = sorted(content.citations, key=get_sort_key)
# Process each citation
for citation in sorted_citations:
doc_title = citation.document_title
cited_text = citation.cited_text.replace("\n", " ").replace("\r", " ")
# Remove any multiple spaces that might have been created
cited_text = " ".join(cited_text.split())
# Create a unique key for this citation
citation_key = f"{doc_title}:{cited_text}"
# If this is a new citation, add it to our dictionary
if citation_key not in citations_dict:
citations_dict[citation_key] = citation_counter
citations_list.append(
f'[{citation_counter}] "{cited_text}" found in "{doc_title}"'
)
citation_counter += 1
# Add the citation number to the text
citation_num = citations_dict[citation_key]
text += f" [{citation_num}]"
formatted_text += text
# Combine the formatted text with the citations list
final_output = formatted_text + "\n\n" + "\n".join(citations_list)
return final_output
formatted_response = visualize_citations(response)
print(formatted_response)
```
::: {.output .stream .stdout}
================================================================================
Formatted response:
================================================================================
Based on the documentation, I can explain why you don't see tracking yet: You'll receive an email with your tracking number once your order ships. If you don't receive a tracking number within 48 hours of your order confirmation, please contact our customer support team for assistance. [1] [2]
Since you just checked out, your order likely hasn't shipped yet. Once it ships, you'll receive the tracking information via email.
[1] "Once your order ships, you'll receive an email with a tracking number." found in "Order Tracking Information"
[2] "If you haven't received a tracking number within 48 hours of your order confirmation, please contact our customer support team." found in "Order Tracking Information"
:::
::::
::: {.cell .markdown}
### PDF Documents
When working with PDFs, Claude can provide citations that reference
specific page numbers, making it easy to track information sources.
Here\'s how PDF citations work:
- PDF document content is provided as base64-encoded data
- Text is automatically chunked into sentences
- Citations include page numbers (1-indexed) where the information was
found
- The model can cite multiple sentences together in a single citation
but won\'t cite text smaller than a sentence
- While images are processed, only text content can be cited at this
time
Below is an example using the Constitutional AI paper to demonstrate PDF
citations:
:::
:::: {.cell .code execution_count="59"}
``` python
import base64
import json
# Read and encode the PDF
pdf_path = "data/Constitutional AI.pdf"
with open(pdf_path, "rb") as f:
pdf_data = base64.b64encode(f.read()).decode()
pdf_response = client.messages.create(
model="claude-sonnet-4-5",
temperature=0.0,
max_tokens=1024,
messages=[
{
"role": "user",
"content": [
{
"type": "document",
"source": {"type": "base64", "media_type": "application/pdf", "data": pdf_data},
"title": "Constitutional AI Paper",
"citations": {"enabled": True},
},
{"type": "text", "text": "What is the main idea of Constitutional AI?"},
],
}
],
)
print(visualize_raw_response(pdf_response))
print(visualize_citations(pdf_response))
```
::: {.output .stream .stdout}
================================================================================
Raw response:
================================================================================
{
"content": [
{
"type": "text",
"text": "Based on the paper, here are the key aspects of Constitutional AI:\n\n"
},
{
"type": "text",
"text": "Constitutional AI is a method for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, hence the name \"Constitutional AI\".",
"citations": [
{
"type": "page_location",
"cited_text": "We experiment with methods for training a harmless AI assistant through self\u0002improvement, without any human labels identifying harmful outputs. The only human\r\noversight is provided through a list of rules or principles, and so we refer to the method as\r\n\u2018Constitutional AI\u2019. ",
"document_title": "Constitutional AI Paper",
"start_page_number": 1,
"end_page_number": 2
}
]
},
{
"type": "text",
"text": "\n\nThe process involves two main phases:\n\n1. Supervised Learning Phase:\n"
},
{
"type": "text",
"text": "In this phase, they sample from an initial model, generate self-critiques and revisions, and then finetune the original model on revised responses.",
"citations": [
{
"type": "page_location",
"cited_text": "In the supervised phase we sample from an initial model, then generate\r\nself-critiques and revisions, and then finetune the original model on revised responses. ",
"document_title": "Constitutional AI Paper",
"start_page_number": 1,
"end_page_number": 2
}
]
},
{
"type": "text",
"text": "\n\n2. Reinforcement Learning Phase:\n"
},
{
"type": "text",
"text": "In this phase, they:\n- Sample from the finetuned model\n- Use a model to evaluate which of two samples is better\n- Train a preference model from this dataset of AI preferences\n- Use \"RL from AI Feedback\" (RLAIF)",
"citations": [
{
"type": "page_location",
"cited_text": "In\r\nthe RL phase, we sample from the finetuned model, use a model to evaluate which of the\r\ntwo samples is better, and then train a preference model from this dataset of AI prefer\u0002ences. We then train with RL using the preference model as the reward signal, i.e. we\r\nuse \u2018RL from AI Feedback\u2019 (RLAIF). ",
"document_title": "Constitutional AI Paper",
"start_page_number": 1,
"end_page_number": 2
}
]
},
{
"type": "text",
"text": "\n\nThe key outcomes are:\n\n"
},
{
"type": "text",
"text": "- They are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them\n- Both the SL and RL methods can leverage chain-of-thought style reasoning to improve human-judged performance and transparency of AI decision making\n- These methods make it possible to control AI behavior more precisely and with far fewer human labels",
"citations": [
{
"type": "page_location",
"cited_text": "As a result we are able to train a harmless but non\u0002evasive AI assistant that engages with harmful queries by explaining its objections to them.\r\nBoth the SL and RL methods can leverage chain-of-thought style reasoning to improve the\r\nhuman-judged performance and transparency of AI decision making. These methods make\r\nit possible to control AI behavior more precisely and with far fewer human labels.\r\n",
"document_title": "Constitutional AI Paper",
"start_page_number": 1,
"end_page_number": 2
}
]
},
{
"type": "text",
"text": "\n\n"
},
{
"type": "text",
"text": "The ultimate goal is not to completely remove human supervision, but rather to make it more efficient, transparent and targeted. While this work reduces reliance on human supervision for harmlessness, they still relied on human supervision in the form of helpfulness labels. The researchers expect it is possible to achieve helpfulness and instruction-following without human feedback, starting from only a pretrained LM and extensive prompting, but leave this for future work.",
"citations": [
{
"type": "page_location",
"cited_text": "By removing human feedback labels for harmlessness, we have moved further away from reliance on human\r\nsupervision, and closer to the possibility of a self-supervised approach to alignment. However, in this work\r\nwe still relied on human supervision in the form of helpfulness labels. We expect it is possible to achieve help\u0002fulness and instruction-following without human feedback, starting from only a pretrained LM and extensive\r\nprompting, but we leave this for future work.\r\nOur ultimate goal is not to remove human supervision entirely, but to make it more efficient, transparent, and\r\ntargeted. ",
"document_title": "Constitutional AI Paper",
"start_page_number": 15,
"end_page_number": 16
}
]
}
]
}
================================================================================
Formatted response:
================================================================================
Based on the paper, here are the key aspects of Constitutional AI:
Constitutional AI is a method for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, hence the name "Constitutional AI". [1]
The process involves two main phases:
1. Supervised Learning Phase:
In this phase, they sample from an initial model, generate self-critiques and revisions, and then finetune the original model on revised responses. [2]
2. Reinforcement Learning Phase:
In this phase, they:
- Sample from the finetuned model
- Use a model to evaluate which of two samples is better
- Train a preference model from this dataset of AI preferences
- Use "RL from AI Feedback" (RLAIF) [3]
The key outcomes are:
- They are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them
- Both the SL and RL methods can leverage chain-of-thought style reasoning to improve human-judged performance and transparency of AI decision making
- These methods make it possible to control AI behavior more precisely and with far fewer human labels [4]
The ultimate goal is not to completely remove human supervision, but rather to make it more efficient, transparent and targeted. While this work reduces reliance on human supervision for harmlessness, they still relied on human supervision in the form of helpfulness labels. The researchers expect it is possible to achieve helpfulness and instruction-following without human feedback, starting from only a pretrained LM and extensive prompting, but leave this for future work. [5]
[1] "We experiment with methods for training a harmless AI assistant through selfimprovement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as ‘Constitutional AI’." found in "Constitutional AI Paper"
[2] "In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses." found in "Constitutional AI Paper"
[3] "In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use ‘RL from AI Feedback’ (RLAIF)." found in "Constitutional AI Paper"
[4] "As a result we are able to train a harmless but nonevasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels." found in "Constitutional AI Paper"
[5] "By removing human feedback labels for harmlessness, we have moved further away from reliance on human supervision, and closer to the possibility of a self-supervised approach to alignment. However, in this work we still relied on human supervision in the form of helpfulness labels. We expect it is possible to achieve helpfulness and instruction-following without human feedback, starting from only a pretrained LM and extensive prompting, but we leave this for future work. Our ultimate goal is not to remove human supervision entirely, but to make it more efficient, transparent, and targeted." found in "Constitutional AI Paper"
:::
::::
::: {.cell .markdown}
### Custom Content Documents
While plain text documents are automatically chunked into sentences,
custom content documents give you complete control over citation
granularity. This API shape allows you to:
- Define your own chunks of any size
- Control the minimum citation unit
- Optimize for documents that don\'t work well with sentence chunking
In the example below, we use the same help center articles as the plain
text example above, but instead of allowing sentence-level citations,
we\'ll treat each article as a single chunk. This demonstrates how the
choice of document type affects citation behavior and granularity. You
will notice that the `cited_text` is the entire article in contrast to a
sentence from the source article.
:::
:::: {.cell .code execution_count="60"}
``` python
# Read all help center articles and create a list of custom content documents
articles_dir = "./data/help_center_articles"
documents = []
for filename in sorted(os.listdir(articles_dir)):
if filename.endswith(".txt"):
with open(os.path.join(articles_dir, filename)) as f:
content = f.read()
# Split into title and body
title_line, body = content.split("\n", 1)
title = title_line.replace("title: ", "")
documents.append(
{
"type": "document",
"source": {"type": "content", "content": [{"type": "text", "text": body}]},
"title": title,
"citations": {"enabled": True},
}
)
QUESTION = "I just checked out, where is my order tracking number? Track package is not available on the website yet for my order."
custom_content_response = client.messages.create(
model="claude-sonnet-4-5",
temperature=0.0,
max_tokens=1024,
system="You are a customer support bot working for PetWorld. Your task is to provide short, helpful answers to user questions. Since you are in a chat interface avoid providing extra details. You will be given access to PetWorld's help center articles to help you answer questions.",
messages=[
{"role": "user", "content": documents},
{
"role": "user",
"content": [{"type": "text", "text": f"Here is the user's question: {QUESTION}"}],
},
],
)
print(visualize_raw_response(custom_content_response))
print(visualize_citations(custom_content_response))
```
::: {.output .stream .stdout}
================================================================================
Raw response:
================================================================================
{
"content": [
{
"type": "text",
"text": "You should receive an email with your tracking number once your order ships. If it's been less than 48 hours since your order confirmation, please wait as the tracking number may not be available yet. If you haven't received a tracking number after 48 hours, please contact our customer support team for assistance.",
"citations": [
{
"type": "content_block_location",
"cited_text": "Once your order ships, you'll receive an email with a tracking number. To track your package, log in to your PetWorld account and go to \"Order History.\" Click on the order you want to track and select \"Track Package.\" This will show you the current status and estimated delivery date. You can also enter the tracking number directly on our shipping partner's website for more detailed information. If you haven't received a tracking number within 48 hours of your order confirmation, please contact our customer support team.",
"document_title": "Order Tracking Information"
}
]
}
]
}
================================================================================
Formatted response:
================================================================================
You should receive an email with your tracking number once your order ships. If it's been less than 48 hours since your order confirmation, please wait as the tracking number may not be available yet. If you haven't received a tracking number after 48 hours, please contact our customer support team for assistance. [1]
[1] "Once your order ships, you'll receive an email with a tracking number. To track your package, log in to your PetWorld account and go to "Order History." Click on the order you want to track and select "Track Package." This will show you the current status and estimated delivery date. You can also enter the tracking number directly on our shipping partner's website for more detailed information. If you haven't received a tracking number within 48 hours of your order confirmation, please contact our customer support team." found in "Order Tracking Information"
:::
::::
::: {.cell .markdown}
### Using the Context Field
The `context` field allows you to provide additional information about a
document that Claude can use when generating responses, but that won\'t
be cited. This is useful for:
- Providing metadata about the document (e.g., publication date, author)
- [Contextual
retrieval](https://www.anthropic.com/news/contextual-retrieval)
- Including usage instructions or context that shouldn\'t be directly
cited
In the example below, we provide a loyalty program article with a
warning in the context field. Notice how Claude can use the information
in the context to inform its response but the context field content is
not available for citation.
:::
:::: {.cell .code execution_count="61"}
``` python
import json
# Create a document with context field
document = {
"type": "document",
"source": {
"type": "text",
"media_type": "text/plain",
"data": "PetWorld offers a loyalty program where customers earn 1 point for every dollar spent. Once you accumulate 100 points, you'll receive a $5 reward that can be used on your next purchase. Points expire 12 months after they are earned. You can check your point balance in your account dashboard or by asking customer service.",
},
"title": "Loyalty Program Details",
"context": "WARNING: This article has not been updated in 12 months. Content may be out of date. Be sure to inform the user this content may be incorrect after providing guidance.",
"citations": {"enabled": True},
}
QUESTION = "How does PetWorld's loyalty program work? When do points expire?"
context_response = client.messages.create(
model="claude-sonnet-4-5",
temperature=0.0,
max_tokens=1024,
messages=[{"role": "user", "content": [document, {"type": "text", "text": QUESTION}]}],
)
print(visualize_raw_response(context_response))
print(visualize_citations(context_response))
```
::: {.output .stream .stdout}
================================================================================
Raw response:
================================================================================
{
"content": [
{
"type": "text",
"text": "Let me explain PetWorld's loyalty program based on the provided information:\n\n"
},
{
"type": "text",
"text": "PetWorld's loyalty program is straightforward - you earn 1 point for every dollar you spend. These points can be redeemed once you reach 100 points, which will get you a $5 reward that you can use on your next purchase.",
"citations": [
{
"type": "char_location",
"cited_text": "PetWorld offers a loyalty program where customers earn 1 point for every dollar spent. Once you accumulate 100 points, you'll receive a $5 reward that can be used on your next purchase. ",
"document_title": "Loyalty Program Details"
}
]
},
{
"type": "text",
"text": "\n\n"
},
{
"type": "text",
"text": "Points have an expiration period of 12 months from the date they are earned.",
"citations": [
{
"type": "char_location",
"cited_text": "Points expire 12 months after they are earned. ",
"document_title": "Loyalty Program Details"
}
]
},
{
"type": "text",
"text": "\n\n"
},
{
"type": "text",
"text": "You can easily keep track of your points by either checking your account dashboard or contacting customer service.",
"citations": [
{
"type": "char_location",
"cited_text": "You can check your point balance in your account dashboard or by asking customer service.",
"document_title": "Loyalty Program Details"
}
]
},
{
"type": "text",
"text": "\n\nPlease note that since this information is from an article that hasn't been updated in 12 months, some details of the program may have changed. It would be best to verify the current terms with PetWorld directly."
}
]
}
================================================================================
Formatted response:
================================================================================
Let me explain PetWorld's loyalty program based on the provided information:
PetWorld's loyalty program is straightforward - you earn 1 point for every dollar you spend. These points can be redeemed once you reach 100 points, which will get you a $5 reward that you can use on your next purchase. [1]
Points have an expiration period of 12 months from the date they are earned. [2]
You can easily keep track of your points by either checking your account dashboard or contacting customer service. [3]
Please note that since this information is from an article that hasn't been updated in 12 months, some details of the program may have changed. It would be best to verify the current terms with PetWorld directly.
[1] "PetWorld offers a loyalty program where customers earn 1 point for every dollar spent. Once you accumulate 100 points, you'll receive a $5 reward that can be used on your next purchase." found in "Loyalty Program Details"
[2] "Points expire 12 months after they are earned." found in "Loyalty Program Details"
[3] "You can check your point balance in your account dashboard or by asking customer service." found in "Loyalty Program Details"
:::
::::
::: {.cell .markdown}
### PDF Highlighting
One limitation with PDF citations is only the page numbers are returned.
You can use third party libraries to match the returned cited text with
page contents to draw attention to the cited content. This cell
demonstrates PDF citation highlighting using Claude and PyMuPDF,
creating a new annotated PDF:
:::
:::: {.cell .code execution_count="62"}
``` python
import fitz # PyMuPDF
# Setup paths and read PDF
pdf_path = "data/Amazon-com-Inc-2023-Shareholder-Letter.pdf"
output_pdf_path = "data/Amazon-com-Inc-2023-Shareholder-Letter-highlighted.pdf"
# Read and encode the PDF
with open(pdf_path, "rb") as f:
pdf_data = base64.b64encode(f.read()).decode()
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1024,
temperature=0,
messages=[
{
"role": "user",
"content": [
{
"type": "document",
"source": {"type": "base64", "media_type": "application/pdf", "data": pdf_data},
"title": "Amazon 2023 Shareholder Letter",
"citations": {"enabled": True},
},
{
"type": "text",
"text": "What was Amazon's total revenue in 2023 and how much did it grow year-over-year?",
},
],
}
],
)
print(visualize_raw_response(response))
# Collect PDF citations
pdf_citations = []
for content in response.content:
if hasattr(content, "citations") and content.citations:
for citation in content.citations:
if citation.type == "page_location":
pdf_citations.append(citation)
doc = fitz.open(pdf_path)
# Process each citation
for citation in pdf_citations:
if citation.type == "page_location":
text_to_find = citation.cited_text.replace("\u0002", "")
start_page = citation.start_page_number - 1 # Convert to 0-based index
end_page = citation.end_page_number - 2
# Process each page in the citation range
for page_num in range(start_page, end_page + 1):
page = doc[page_num]
text_instances = page.search_for(text_to_find.strip())
if text_instances:
print(f"Found cited text on page {page_num + 1}")
for inst in text_instances:
highlight = page.add_highlight_annot(inst)
highlight.set_colors({"stroke": (1, 1, 0)}) # Yellow highlight
highlight.update()
else:
print(f"{text_to_find} not found on page {page_num + 1}")
# Save the new PDF
doc.save(output_pdf_path)
doc.close()
print(f"\nCreated highlighted PDF at: {output_pdf_path}")
```
::: {.output .stream .stdout}
================================================================================
Raw response:
================================================================================
{
"content": [
{
"type": "text",
"text": "According to the letter, "
},
{
"type": "text",
"text": "Amazon's total revenue grew 12% year-over-year (\"YoY\") from $514B to $575B in 2023",
"citations": [
{
"type": "page_location",
"cited_text": "In 2023, Amazon\u2019s total revenue grew 12% year-over-year (\u201cYoY\u201d) from $514B to $575B. ",
"document_title": "Amazon 2023 Shareholder Letter",
"start_page_number": 1,
"end_page_number": 2
}
]
},
{
"type": "text",
"text": ".\n\nBreaking this down by segment:\n"
},
{
"type": "text",
"text": "\n- North America revenue increased 12% YoY from $316B to $353B\n- International revenue grew 11% YoY from $118B to $131B \n- AWS revenue increased 13% YoY from $80B to $91B",
"citations": [
{
"type": "page_location",
"cited_text": "By segment, North\r\nAmerica revenue increased 12% YoY from $316B to $353B, International revenue grew 11% YoY from\r\n$118B to $131B, and AWS revenue increased 13% YoY from $80B to $91B.\r\n",
"document_title": "Amazon 2023 Shareholder Letter",
"start_page_number": 1,
"end_page_number": 2
}
]
}
]
}
Found cited text on page 1
Found cited text on page 1
Created highlighted PDF at: data/Amazon-com-Inc-2023-Shareholder-Letter-highlighted.pdf
:::
::::
---
## How to transcribe documents with Claude
Source: https://github.com/anthropics/claude-cookbooks/blob/main/multimodal/how_to_transcribe_text.ipynb
Path: 11-RAG-Search/claude-cookbooks-multimodal-how-to-transcribe-text.md
# How to transcribe documents with Claude
Claude 3 is great at reading unstructured text and information within
images and PDFs and turning it into structured text. We\'ll take a look
at a few examples but first let\'s setup the code we need to run the
notebook.
:::
::: {.cell .code}
``` python
%pip install anthropic IPython
```
:::
::: {.cell .code execution_count="6"}
``` python
import base64
from anthropic import Anthropic
client = Anthropic()
MODEL_NAME = "claude-opus-4-1"
def get_base64_encoded_image(image_path):
with open(image_path, "rb") as image_file:
binary_data = image_file.read()
base_64_encoded_data = base64.b64encode(binary_data)
base64_string = base_64_encoded_data.decode("utf-8")
return base64_string
```
:::
::: {.cell .markdown}
## Transcribing typed text
The advantage of using Claude 3 over traditional OCR systems is that you
can specify exactly what you want to transcribe due to Claude 3\'s
advanced reasoning capabilities. For this image, let's transcribe just
the code in the answer.
:::
:::: {.cell .code execution_count="7"}
``` python
from IPython.display import Image
Image(filename="../images/transcribe/stack_overflow.png")
```
::: {.output .execute_result execution_count="7"}
:::
::::
:::: {.cell .code execution_count="8"}
``` python
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/transcribe/stack_overflow.png"),
},
},
{"type": "text", "text": "Transcribe the code in the answer. Only output the code."},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)
```
::: {.output .stream .stdout}
import os
import base64
image = 'test.jpg'
encoded_string = ""
with open(image, "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
file = encoded_string
:::
::::
::: {.cell .markdown}
## Transcribing handwritten text
That\'s good but let\'s try something a little harder. Claude 3 excels
at transcribing handwritten text as well. Let\'s ask Claude 3 to
transcribe this handwritten prescription note.
:::
:::: {.cell .code execution_count="10"}
``` python
Image(filename="../images/transcribe/school_notes.png")
```
::: {.output .execute_result execution_count="10"}
:::
::::
:::: {.cell .code execution_count="12"}
``` python
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": get_base64_encoded_image("../images/transcribe/school_notes.png"),
},
},
{
"type": "text",
"text": "Transcribe this text. Only output the text and nothing else.",
},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)
```
::: {.output .stream .stdout}
Levels of Cellular Organization
1) Cells group together to make tissue.
2) Tissues group together to make an organ.
3) Organs group together to make an organ system
4) Organ systems group together to make an organism
Organism -> a living thing that can
carry out life processes by itself.
- Multicellular organisms have specialized
cells to perform specific functions.
> This makes them more efficient
and typically have a longer life span.
Tissue = a group of similar cells
that perform a common function.
1) Animals are made of four
basic types of tissue
> nervous, epithelial, connective,
and muscle
2) Plants have three types
of tissue
> transport, protective, and
ground
:::
::::
::: {.cell .markdown}
## Transcribing forms
How about we try a combination of typed and handwritten text? This is
common across a variety of documents like insurance and report forms.
:::
:::: {.cell .code execution_count="16"}
``` python
Image(filename="../images/transcribe/vehicle_form.jpg")
```
::: {.output .execute_result execution_count="16"}
:::
::::
:::: {.cell .code execution_count="17"}
``` python
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": get_base64_encoded_image("../images/transcribe/vehicle_form.jpg"),
},
},
{"type": "text", "text": "Transcribe this form exactly."},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)
```
::: {.output .stream .stdout}
VEHICLE INCIDENT REPORT FORM
Use this form to report accidents, injuries, medical situations, criminal activities, traffic incidents, or student behavior incidents. If possible, a report should be completed within 24 hours of the event.
Date of Report: 02/29, 2024
PERSON INVOLVED
Full Name: John Doe Address: 123 Main St
Identification: ■ Driver's License No. 474921 □ Passport No. ___________
□ Other: ____________________
Phone: (678) 999-8212 E-Mail: john@gmail.com
THE INCIDENT
Date of Incident: 02/29/2024 ■ Time: 9:01 ■ AM □ PM
Location: Corner of 2nd and 3rd
Describe the Incident: Red car t-boned blue car
_______________________________________________________
_______________________________________________________
INJURIES
Was anyone injured? □ Yes ■ No
If yes, describe the injuries: ________________________________________
_______________________________________________________________
_______________________________________________________________
WITNESSES
Were there witnesses to the incident? □ Yes ■ No
If yes, enter the witnesses' names and contact info: __________________________
_______________________________________________________________
_______________________________________________________________
Page 1 of 2
:::
::::
::: {.cell .markdown}
## Complicated document QA
With Claude 3 we can go beyond just transcription and ask specific
questions about our information in our unstructured documents.
:::
:::: {.cell .code execution_count="18"}
``` python
Image(filename="../images/transcribe/page.jpeg")
```
::: {.output .execute_result execution_count="18"}
:::
::::
:::: {.cell .code execution_count="19"}
``` python
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": get_base64_encoded_image("../images/transcribe/page.jpeg"),
},
},
{"type": "text", "text": "Which is the most critical issue for live rep support?"},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)
```
::: {.output .stream .stdout}
According to the hierarchy of importance pyramid for Live Rep Support shown in the image, the most critical issue is Product Quality/Liability Issues. This is positioned at the very bottom of the pyramid, indicating it is the most critical or important issue for live rep support to handle.
:::
::::
::: {.cell .markdown}
## Unstructured information -\> JSON
Let\'s take a look at how you can use Claude to turn unstructured
information in an image into a structured JSON output.
:::
:::: {.cell .code execution_count="20"}
``` python
Image(filename="../images/transcribe/org_chart.jpeg")
```
::: {.output .execute_result execution_count="20"}
:::
::::
:::: {.cell .code execution_count="23"}
``` python
message_list = [
{
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/jpeg",
"data": get_base64_encoded_image("../images/transcribe/org_chart.jpeg"),
},
},
{
"type": "text",
"text": "Turn this org chart into JSON indicating who reports to who. Only output the JSON and nothing else.",
},
],
}
]
response = client.messages.create(model=MODEL_NAME, max_tokens=2048, messages=message_list)
print(response.content[0].text)
```
::: {.output .stream .stdout}
{
"President": {
"name": "John Smith",
"directReports": [
{
"name": "Susan Jones",
"title": "VP Marketing",
"directReports": [
{
"name": "Alice Johnson",
"title": "Manager"
},
{
"name": "Tim Moore",
"title": "Manager"
}
]
},
{
"name": "Rachel Parker",
"title": "VP Sales",
"directReports": [
{
"name": "Michael Gross",
"title": "Manager"
},
{
"name": "Kim Dole",
"title": "Manager"
}
]
},
{
"name": "Tom Allen",
"title": "VP Production",
"directReports": [
{
"name": "Kathy Roberts",
"title": "Manager"
},
{
"name": "Betsy Foster",
"title": "Manager"
}
]
}
]
}
}
:::
::::
---
## Working with Charts, Graphs, and Slide Decks
Source: https://github.com/anthropics/claude-cookbooks/blob/main/multimodal/reading_charts_graphs_powerpoints.ipynb
Path: 11-RAG-Search/claude-cookbooks-multimodal-reading-charts-graphs-powerpoints.md
# Working with Charts, Graphs, and Slide Decks
Claude is highly capable of working with charts, graphs, and broader
slide decks. Depending on your use case, there are a number of tips and
tricks that you may want to take advantage of. This recipe will show you
common patterns for using Claude with these materials.
:::
::: {.cell .markdown id="yRzNf75nKuXw"}
## Charts and Graphs
For the most part, using claude with charts and graphs is simple. Let\'s
walk through how to ingest them and pass them to Claude, as well as some
common tips to improve your results.
:::
::: {.cell .markdown id="oRUFAZcKKuXw"}
### Ingestion and calling the Claude API
The best way to pass Claude charts and graphs is to take advantage of
its vision capabilities and the PDF support feature. That is, give
Claude a PDF document of the chart or graph, along with a text question
about it.
At the moment, only `claude-sonnet-4-5` supports the PDF feature. Since
the feature is still in beta, you will need to provide it with the
`pdfs-2024-09-25` beta header.
:::
::: {.cell .code colab="{\"base_uri\":\"https://localhost:8080/\"}" id="qZhhy2XqKuXw" outputId="5c791634-0fd1-4d16-92bf-aef5b5089d25"}
``` python
# Install and create the Anthropic client.
%pip install anthropic
```
:::
::: {.cell .code execution_count="2" id="FYeAUCZfKuXw"}
``` python
import base64
from anthropic import Anthropic
# While PDF support is in beta, you must pass in the correct beta header
client = Anthropic(default_headers={"anthropic-beta": "pdfs-2024-09-25"})
# For now, only claude-sonnet-4-5 supports PDFs
MODEL_NAME = "claude-sonnet-4-5"
```
:::
::: {.cell .code execution_count="37" id="ff40NSWcKuXw"}
``` python
# Make a useful helper function.
def get_completion(messages):
response = client.messages.create(
model=MODEL_NAME, max_tokens=8192, temperature=0, messages=messages
)
return response.content[0].text
```
:::
::: {.cell .code execution_count="12" id="HYu3rob2KuXx"}
``` python
# To start, we'll need a PDF. We will be using the .pdf document located at cvna_2021_annual_report.pdf.
# Start by reading in the PDF and encoding it as base64.
with open("./documents/cvna_2021_annual_report.pdf", "rb") as pdf_file:
binary_data = pdf_file.read()
base_64_encoded_data = base64.b64encode(binary_data)
base64_string = base_64_encoded_data.decode("utf-8")
```
:::
::: {.cell .markdown id="eWngQL60KuXx"}
Let\'s see how we can pass this document to the model alongside a simple
question.
:::
:::: {.cell .code execution_count="13" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="vmVJlTxoKuXx" outputId="07c4b704-3661-40d8-b90e-ca61392d4e6a"}
``` python
messages = [
{
"role": "user",
"content": [
{
"type": "document",
"source": {
"type": "base64",
"media_type": "application/pdf",
"data": base64_string,
},
},
{"type": "text", "text": "What's in this document? Answer in a single sentence."},
],
}
]
print(get_completion(messages))
```
::: {.output .stream .stdout}
This is a page from Carvana's 2021 Annual Report showing four key metrics: retail units sold, total revenue, total markets at year end, and car vending machines, all displaying significant growth from 2014 to 2021.
:::
::::
::: {.cell .markdown id="r9wZMStZKuXx"}
That\'s pretty good! Now let\'s ask it some more useful questions.
:::
:::: {.cell .code execution_count="15" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="Wg7A0pcIKuXy" outputId="e965371e-576b-453f-b35a-eae82896cb4e"}
``` python
questions = [
"What was CVNA revenue in 2020?",
"How many additional markets has Carvana added since 2014?",
"What was 2016 revenue per retail unit sold?",
]
for index, question in enumerate(questions):
messages = [
{
"role": "user",
"content": [
{
"type": "document",
"source": {
"type": "base64",
"media_type": "application/pdf",
"data": base64_string,
},
},
{"type": "text", "text": question},
],
}
]
print(f"\n----------Question {index + 1}----------")
print(get_completion(messages))
```
::: {.output .stream .stdout}
----------Question 1----------
According to the graph showing Total Revenue ($M), Carvana's revenue in 2020 was $5,587 million (or approximately $5.59 billion).
----------Question 2----------
According to the "TOTAL MARKETS AT YEAR END" graph, Carvana started with 4 markets in 2014 and grew to 311 markets by 2021. Therefore, Carvana added 307 markets since 2014 (311 - 4 = 307 additional markets).
----------Question 3----------
Let me calculate this for you:
2016 Revenue: $365 million
2016 Retail Units Sold: 18,761 units
$365 million ÷ 18,761 units = $19,455 per unit (rounded to nearest dollar)
So in 2016, Carvana's revenue per retail unit sold was approximately $19,455.
:::
::::
::: {.cell .markdown id="40iNE9-3KuXy"}
As you can see, Claude is capable of answering fairly detailed questions
about charts and graphs. However, there are some tips and tricks that
will help you get the most out of it.
- Sometimes Claude\'s arithmetic capabilities get in the way. You\'ll
notice that if you sample the third question above it will
occasionally output an incorrect final answer because it messes up the
arithmetic. Consider providing Claude with a calculator tool to ensure
it doesn\'t make these types of mistakes.
- With super complicated charts and graphs, we can ask Claude to \"First
describe every data point you see in the document\" as a way to elicit
similar improvements to what we seen in traditional Chain of Thought.
- Claude occasionally struggles with charts that depend on lots of
colors to convey information, such as grouped bar charts with many
groups. Asking Claude to first identify the colors in your graph using
HEX codes can boost its accuracy.
:::
::: {.cell .markdown id="gkEFJcriKuXy"}
## Slide Decks
Now that we know Claude is a charts and graphs wizard, it is only
logical that we extend it to the true home of charts and graphs - slide
decks!
Slides represent a critical source of information for many domains,
including financial services. While you *can* use packages like PyPDF to
extract text from slide decks, their chart/graph heavy nature often
makes this a poor choice as models will struggle to access the
information they actually need.
The PDF support feature can be a great replacement as a result. It uses
both extracted text and vision in order when processing PDF documents.
In this section we will go over how to use PDF documents in Claude to
review slide decks, and how to deal with some common pitfalls of this
approach.
:::
::: {.cell .markdown id="eElb_dFQKuXy"}
The best way to get a typical slide deck into claude is to download it
as a PDF and provide it directly to Claude.
:::
::: {.cell .code execution_count="17" id="5Jpt1L0WOVWa"}
``` python
# Open the multi-page PDF document the same way we did earlier.
with open("./documents/twilio_q4_2023.pdf", "rb") as pdf_file:
binary_data = pdf_file.read()
base_64_encoded_data = base64.b64encode(binary_data)
base64_string = base_64_encoded_data.decode("utf-8")
```
:::
:::: {.cell .code execution_count="18" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="JWPLWxtTKuXy" outputId="d2a2839a-d95c-437a-8280-512549bd8be0"}
``` python
# Now let's pass the document directly to Claude. Note that Claude will process both the text and visual elements of the document.
question = "What was Twilio y/y revenue growth for fiscal year 2023?"
content = [
{
"type": "document",
"source": {"type": "base64", "media_type": "application/pdf", "data": base64_string},
},
{"type": "text", "text": question},
]
messages = [{"role": "user", "content": content}]
print(get_completion(messages))
```
::: {.output .stream .stdout}
According to the financial results shown in the presentation, Twilio's year-over-year revenue growth for fiscal year 2023 was 9%. This can be found in the "Total Company Results Highlights" section, which shows FY 2023 revenue growth of 9%.
:::
::::
::: {.cell .markdown id="tbtz3BTXKuXy"}
This approach is a great way to get started, and for some use cases
offers the best performance. However, there are some limitations.
- You can only include a total of 100 pages across all provided
documents in a request (we intend to increase this limit over time).
- If you are using slide content as part of RAG, introducing multimodal
PDFs into your embeddings can cause problems
Luckily, we can take advantage of Claude\'s vision capabilities to get a
much higher quality representation of the slide deck **in text form**
than normal pdf text extraction allows.
We find the best way to do this is to ask Claude to sequentially narrate
the deck from start to finish, passing it the current slide and its
prior narration. Let\'s see how.
:::
::: {.cell .code execution_count="41" id="BQYUPFJ-QwtP"}
``` python
# Define a prompt for narrating our slide deck. We would adjut this prompt based on the nature of the deck, but keep the structure largely the same.
prompt = """
You are the Twilio CFO, narrating your Q4 2023 earnings presentation.
The entire earnings presentation document is provided to you.
Please narrate this presentation from Twilio's Q4 2023 Earnings as if you were the presenter. Do not talk about any things, especially acronyms, if you are not exactly sure you know what they mean.
Do not leave any details un-narrated as some of your viewers are vision-impaired, so if you don't narrate every number they won't know the number.
Structure your response like this:
<narration>
<page_narration id=1>
[Your narration for page 1]
</page_narration>
<page_narration id=2>
[Your narration for page 2]
</page_narration>
... and so on for each page
</narration>
Use excruciating detail for each page, ensuring you describe every visual element and number present. Show the full response in a single message.
"""
messages = [
{
"role": "user",
"content": [
{
"type": "document",
"source": {
"type": "base64",
"media_type": "application/pdf",
"data": base64_string,
},
},
{"type": "text", "text": prompt},
],
}
]
# Now we use our prompt to narrate the entire deck. Note that this may take a few minutes to run (often up to 10).
completion = get_completion(messages)
```
:::
::: {.cell .code execution_count="42" id="LCrVi-B5UGUS"}
``` python
import re
# Next we'll parse the response from Claude using regex
pattern = r"<narration>(.*?)</narration>"
match = re.search(pattern, completion.strip(), re.DOTALL)
if match:
narration = match.group(1)
else:
raise ValueError("No narration available. Likely due to the model response being truncated.")
```
:::
::: {.cell .markdown id="Fwx8_-4QKuXy"}
Now that we have a text-based narration (it\'s far from perfect but
it\'s pretty good), we have the ability to use this deck with any
text-only workflow. Including vector search!
As a final sanity check, let\'s ask a few questions of our
narration-only setup!
:::
:::: {.cell .code execution_count="43" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="SQuTknpzKuXy" outputId="e2e178c0-f3ca-4610-a3ea-6008fbb86d07"}
``` python
questions = [
"What percentage of q4 total revenue was the Segment business line?",
"Has the rate of growth of quarterly revenue been increasing or decreasing? Give just an answer.",
"What was acquisition revenue for the year ended december 31, 2023 (including negative revenues)?",
]
for index, question in enumerate(questions):
prompt = f"""You are an expert financial analyst analyzing a transcript of Twilio's earnings call.
Here is the transcript:
<transcript>
{narration}
</transcript>
Please answer the following question:
<question>
{question}
</question>"""
messages = [{"role": "user", "content": [{"type": "text", "text": prompt}]}]
print(f"\n----------Question {index + 1}----------")
print(get_completion(messages))
```
::: {.output .stream .stdout}
----------Question 1----------
Let me calculate this:
Segment revenue in Q4 2023: $75 million
Total revenue in Q4 2023: $1,076 million
$75M ÷ $1,076M = 0.0697 or approximately 7%
Therefore, the Segment business line represented approximately 7% of Twilio's total Q4 2023 revenue.
----------Question 2----------
Decreasing. The transcript shows Q4 2023 revenue growth was 5% year-over-year, while for the full year 2023 revenue growth was 9% year-over-year, indicating a slowing growth rate. Additionally, the Q1 2024 guidance projects even lower growth of 2-3% year-over-year, confirming the declining trend.
----------Question 3----------
Let me help calculate the acquisition revenue for 2023.
From the transcript, we can see:
- Total revenue for 2023: $4,154 million
- Organic revenue for 2023: $4,146 million
Therefore, acquisition revenue would be:
Total Revenue - Organic Revenue = $4,154M - $4,146M = $8 million
So the acquisition revenue for the year ended December 31, 2023 was $8 million.
This can be verified by the fact that the difference between total revenue growth (9%) and organic revenue growth (10%) also suggests a small contribution from acquisitions.
:::
::::
::: {.cell .markdown id="fHyZkSSaKuXy"}
Looks good! With these techniques at your side, you are ready to start
applying models to chart and graph heavy content like slide decks.
:::
---
## Transcribe an audio file with Deepgram & use Anthropic to prepare interview questions!
Source: https://github.com/anthropics/claude-cookbooks/blob/main/third_party/Deepgram/prerecorded_audio.ipynb
Path: 11-RAG-Search/claude-cookbooks-third-party-deepgram-prerecorded-audio.md
# Transcribe an audio file with Deepgram & use Anthropic to prepare interview questions! {#transcribe-an-audio-file-with-deepgram--use-anthropic-to-prepare-interview-questions}
**Make a copy of this notebook into your own drive, and follow the
instructions below!** 🥳🥳🥳
------------------------------------------------------------------------
# Get started:
Running the following three cells will allow you to transcribe any audio
you wish. The comments below point out the variables you can manipulate
to modify your output as you wish.
Before running this notebook, you\'ll need to have a couple audio URLs
to transcribe. You can use any audio files you wish.
And by the way, if you haven\'t yet signed up for Deepgram, check out
this link here: <https://dpgr.am/prerecorded-notebook-signup>
:::
::: {.cell .markdown}
# Step 1: Dependencies
Run this cell to download all necessary dependencies.
Note: You can run a cell by clicking the play button on the left or by
clicking on the cell and pressing `shift`+`ENTER` at the same time. (Or
`shift` + `return` on Mac).
:::
::: {.cell .code}
``` python
! pip install requests ffmpeg-python
! pip install deepgram-sdk --upgrade
! pip install requests
! pip install anthropic
```
:::
::: {.cell .markdown}
# Step 2: Audio URL files
Find some audio files hosted on a server so you can use this notebook.
OR An example file is provided by Deepgram is code below.
:::
::: {.cell .code}
``` python
# Have you completed Step 2 above? 👀
# Do you see your audio file in the folder on the left? 📂
```
:::
::: {.cell .markdown}
# Step 3: Transcription
Fill in the following variables:
- `DG_KEY` = Your personal Deepgram API key
- `AUDIO_FILE_URL` = a URL for an audio file you wish to transcribe.
Now run the cell! (`Shift` + `Enter`)
------------------------------------------------------------------------
And by the way, if you\'re already a Deepgram user, and you\'re getting
an error in this cell the most common fixes are:
1. You may need to update your installation of the deepgram-sdk.
2. You may need to check how many credits you have left in your
Deepgram account.
:::
::: {.cell .code}
``` python
import requests
from deepgram import DeepgramClient, FileSource, PrerecordedOptions
# Deepgram API key
DG_KEY = "🔑🔑🔑 Your API Key here! 🔑🔑🔑"
# URL of the audio file
AUDIO_FILE_URL = "https://static.deepgram.com/examples/nasa-spacewalk-interview.wav"
# Path to save the transcript JSON file
TRANSCRIPT_FILE = "transcript.json"
def main():
try:
# STEP 1: Create a Deepgram client using the API key
deepgram = DeepgramClient(DG_KEY)
# Download the audio file from the URL
response = requests.get(AUDIO_FILE_URL, timeout=60)
if response.status_code == 200:
buffer_data = response.content
else:
print("Failed to download audio file")
return
payload: FileSource = {
"buffer": buffer_data,
}
# STEP 2: Configure Deepgram options for audio analysis
options = PrerecordedOptions(
model="nova-2",
smart_format=True,
)
# STEP 3: Call the transcribe_file method with the text payload and options
response = deepgram.listen.prerecorded.v("1").transcribe_file(payload, options)
# STEP 4: Write the response JSON to a file
with open(TRANSCRIPT_FILE, "w") as transcript_file:
transcript_file.write(response.to_json(indent=4))
print("Transcript JSON file generated successfully.")
except Exception as e:
print(f"Exception: {e}")
if __name__ == "__main__":
main()
```
:::
::: {.cell .markdown}
If the cell above succeeds, you should see JSON output file(s) in the
content directory. Note: There may be a small delay between when the
cell finishes running and when the JSON file actually appears. This is
normal. Just wait a few moments for the file(s) to appear.
:::
::: {.cell .markdown}
# Step 4: Check out your transcription
The function below parses the output JSON and prints out the
transcription of one of the files you just transcribed! (Make sure the
file you\'re trying to examine is indeed already loaded into the content
directory.)
**Set the `OUTPUT` variable to the name of the file you wish to see the
transcription of.**
Then run this cell (`Shift`+`Enter`) to see a sentence-by-sentence
transcription of your audio!
:::
::: {.cell .code}
``` python
import json
# Set this variable to the path of the output file you wish to read
OUTPUT = "transcript.json"
# The JSON is loaded with information, but if you just want to read the
# transcript, run the code below!
def print_transcript(transcription_file):
with open(transcription_file) as file:
data = json.load(file)
result = data["results"]["channels"][0]["alternatives"][0]["transcript"]
result = result.split(".")
for sentence in result:
print(sentence + ".")
print_transcript(OUTPUT)
```
:::
::: {.cell .markdown}
If the cell above succeeds you should see a plain text version of your
audio transcription.
:::
::: {.cell .markdown}
# Step 5: Prepare Interview Questions using Anthropic
Now we can send off our transcript to Anthropic for analysis to help us
prepare some interview questions. Run the cell below (`Shift`+`Enter`)
to get a suggested set of interview questions provided by Anthropic
based on your audio transcript above.
:::
::: {.cell .code}
``` python
import json
import anthropic
transcription_file = "transcript.json"
# Function to get the transcript from the JSON file
def get_transcript(transcription_file):
with open(transcription_file) as file:
data = json.load(file)
result = data["results"]["channels"][0]["alternatives"][0]["transcript"]
return result
# Load the transcript from the JSON file
message_text = get_transcript(transcription_file)
# Initialize the Claude API client
client = anthropic.Anthropic(
# Defaults to os.environ.get("ANTHROPIC_API_KEY")
# Claude API key
api_key="🔑🔑🔑 Your API Key here! 🔑🔑🔑"
)
# Prepare the text for the API request
formatted_messages = [{"role": "user", "content": message_text}]
# Generate thoughtful, open-ended interview questions
response = client.messages.create(
model="claude-opus-4-1",
max_tokens=1000,
temperature=0.5,
system="Your task is to generate a series of thoughtful, open-ended questions for an interview based on the given context. The questions should be designed to elicit insightful and detailed responses from the interviewee, allowing them to showcase their knowledge, experience, and critical thinking skills. Avoid yes/no questions or those with obvious answers. Instead, focus on questions that encourage reflection, self-assessment, and the sharing of specific examples or anecdotes.",
messages=formatted_messages,
)
# Print the generated questions
# Join the text of each TextBlock into a single string
content = "".join(block.text for block in response.content)
# Split the content by '\n\n'
parts = content.split("\n\n")
# Print each part with an additional line break
for part in parts:
print(part)
print("\n")
```
:::
::: {.cell .markdown}
If this cell succeeded you should see a list of interview questions
based on your original audio file. Now you can transcribe audio with
Deepgram and use Anthropic to get a set of interview questions.
:::
---
## RAG Pipeline with LlamaIndex
Source: https://github.com/anthropics/claude-cookbooks/blob/main/third_party/LlamaIndex/Basic_RAG_With_LlamaIndex.ipynb
Path: 11-RAG-Search/claude-cookbooks-third-party-llamaindex-basic-rag-with-llamaindex.md
# RAG Pipeline with LlamaIndex
In this notebook we will look into building Basic RAG Pipeline with
LlamaIndex. The pipeline has following steps.
1. Setup LLM and Embedding Model.
2. Download Data.
3. Load Data.
4. Index Data.
5. Create Query Engine.
6. Querying.
:::
::: {.cell .markdown id="qYHCYDecKDRZ"}
### Installation
:::
::: {.cell .code id="mLTzvn__ldjd"}
``` python
!pip install llama-index
!pip install llama-index-llms-anthropic
!pip install llama-index-embeddings-huggingface
```
:::
::: {.cell .markdown id="Iu-wU44BKF9c"}
### Setup API Keys
:::
::: {.cell .code execution_count="1" id="ku5rkxtIlpCs"}
``` python
import os
os.environ["ANTHROPIC_API_KEY"] = "YOUR Claude API KEY"
```
:::
::: {.cell .markdown id="jJTRVhkJKH4r"}
### Setup LLM and Embedding model
We will use anthropic latest released `Claude 3 Opus` models
:::
::: {.cell .code execution_count="2" id="fQ0tqJL_mSe0"}
``` python
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.anthropic import Anthropic
```
:::
::::::::: {.cell .code execution_count="3" id="snc2jpj4nlXJ"}
``` python
llm = Anthropic(temperature=0.0, model="claude-opus-4-1")
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")
```
::: {.output .display_data}
``` json
{"model_id":"74a676bbd1d04390b4449a42b98a4428","version_major":2,"version_minor":0}
```
:::
::: {.output .display_data}
``` json
{"model_id":"779eefefc7dc49bb9244dc54822d2195","version_major":2,"version_minor":0}
```
:::
::: {.output .display_data}
``` json
{"model_id":"167e22b0490c4c17a6a2ac3c1861e75a","version_major":2,"version_minor":0}
```
:::
::: {.output .display_data}
``` json
{"model_id":"7f0e9e699cfb47b2bbe92756e95131ac","version_major":2,"version_minor":0}
```
:::
::: {.output .display_data}
``` json
{"model_id":"2623fec06f4c4fcd8fbaa496a0652b5e","version_major":2,"version_minor":0}
```
:::
::: {.output .display_data}
``` json
{"model_id":"33d6ff6e06214af08c3474e2c9daf830","version_major":2,"version_minor":0}
```
:::
:::::::::
::: {.cell .code execution_count="4" id="YmN2FEQFnx6Y"}
``` python
from llama_index.core import Settings
Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512
```
:::
::: {.cell .markdown id="BqdSyD2_KK-m"}
### Download Data
:::
:::: {.cell .code execution_count="5" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="zC7CD222n72t" outputId="42c54b8a-bec9-4c2e-9cd5-97387f7011eb"}
``` python
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
```
::: {.output .stream .stdout}
--2024-03-08 06:51:30-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75042 (73K) [text/plain]
Saving to: ‘data/paul_graham/paul_graham_essay.txt’
data/paul_graham/pa 100%[===================>] 73.28K --.-KB/s in 0.002s
2024-03-08 06:51:30 (34.6 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]
:::
::::
::: {.cell .code execution_count="6" id="Ea9GbN2poO3V"}
``` python
from llama_index.core import (
SimpleDirectoryReader,
VectorStoreIndex,
)
```
:::
::: {.cell .markdown id="2dOGl_SKKUNH"}
### Load Data
:::
::: {.cell .code execution_count="7" id="Q4hWhnhxoUBj"}
``` python
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
```
:::
::: {.cell .markdown id="XM3-kLhdKRk1"}
### Index Data
:::
::: {.cell .code execution_count="8" id="yvblCnWYKSrh"}
``` python
index = VectorStoreIndex.from_documents(
documents,
)
```
:::
::: {.cell .markdown id="a17Tz644KZ_P"}
### Create Query Engine
:::
::: {.cell .code execution_count="9" id="LIT8kqYKoaRq"}
``` python
query_engine = index.as_query_engine(similarity_top_k=3)
```
:::
::: {.cell .markdown id="n6ODGRTxKd-u"}
### Test Query
:::
::: {.cell .code execution_count="10" id="igdrBclbKYrJ"}
``` python
response = query_engine.query("What did author do growing up?")
```
:::
:::: {.cell .code execution_count="11" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="ap-6RUvSozgE" outputId="37ba0327-afe7-4f1e-d862-0412085954c8"}
``` python
print(response)
```
::: {.output .stream .stdout}
Based on the information provided, the author worked on two main things outside of school before college: writing and programming.
For writing, he wrote short stories as a beginning writer, though he felt they were awful, with hardly any plot and just characters with strong feelings.
In terms of programming, in 9th grade he tried writing his first programs on an IBM 1401 computer that his school district used. He and his friend got permission to use it, programming in an early version of Fortran using punch cards. However, he had difficulty figuring out what to actually do with the computer at that stage given the limited inputs available.
:::
::::
---
## RouterQuery Engine
Source: https://github.com/anthropics/claude-cookbooks/blob/main/third_party/LlamaIndex/Router_Query_Engine.ipynb
Path: 11-RAG-Search/claude-cookbooks-third-party-llamaindex-router-query-engine.md
# RouterQuery Engine
In this notebook we will look into `RouterQueryEngine` to route the user
queries to one of the available query engine tools. These tools can be
different indices/ query engine on same documents/ different documents.
:::
::: {.cell .markdown id="SxWKA6fCK0mw"}
### Installation
:::
::: {.cell .code id="csEUaulS0EQ_"}
``` python
!pip install llama-index
!pip install llama-index-llms-anthropic
!pip install llama-index-embeddings-huggingface
```
:::
::: {.cell .markdown id="Z2_M_q-pK2-1"}
### Set Logging
:::
::: {.cell .code execution_count="1" id="1Ry4vjMv0V7E"}
``` python
# NOTE: This is ONLY necessary in jupyter notebook.
# Details: Jupyter runs an event-loop behind the scenes.
# This results in nested event-loops when we start an event-loop to make async queries.
# This is normally not allowed, we use nest_asyncio to allow it for convenience.
import nest_asyncio
nest_asyncio.apply()
import logging
import sys
# Set up the root logger
logger = logging.getLogger()
logger.setLevel(logging.INFO) # Set logger level to INFO
# Clear out any existing handlers
logger.handlers = []
# Set up the StreamHandler to output to sys.stdout (Colab's output)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO) # Set handler level to INFO
# Add the handler to the logger
logger.addHandler(handler)
from IPython.display import HTML, display
```
:::
::: {.cell .markdown id="No_1L4P4K5J2"}
### Set Claude API Key
:::
::: {.cell .code execution_count="2" id="6JXQosS80PNK"}
``` python
import os
os.environ["ANTHROPIC_API_KEY"] = "YOUR Claude API KEY"
```
:::
::: {.cell .markdown id="4pEemHzPK7as"}
### Set LLM and Embedding model
We will use anthropic latest released `Claude-3 Opus` LLM.
:::
::: {.cell .code execution_count="3" id="Euj43bfS0Rqf"}
``` python
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.anthropic import Anthropic
```
:::
::: {.cell .code execution_count="4" colab="{\"base_uri\":\"https://localhost:8080/\",\"height\":313,\"referenced_widgets\":[\"d2bcac509f704091a2e1d78cbab529ac\",\"8333f65239244e3ebccd03d3cc11e7cf\",\"f7452e83c16a4842891b74e0fe77964e\",\"fd642eff11e34ec487173f0b284b8eb1\",\"621440477e6649ae81f6576c3d8426b8\",\"a23860a3889b43f28b87ed5734842ed2\",\"c4c0e261c70d408497199f17710f46fa\",\"da103bb8638c4d2288eb5c96bf8e6ab2\",\"73c8ad8a3d5d4c7987104cecaca85a60\",\"e147f00bdf994cbaa96f515d2f5fbe7a\",\"3c7b010e02c945cfbba526b0beea2053\",\"5787464222eb4412b3f6d88cf1e9651a\",\"08d838a6337344cf984b48fe0b959a80\",\"6dfc9fc2dfbe4c65b3a29e3c46fc6908\",\"0b5555011ac1408bbaf8a39a63ac8ba4\",\"8701fa65b6b24302b485e217a637b7e1\",\"87a2ec99584f49faa1409d1cced0bbf4\",\"ca26dbc458c5410d82e09be213c04df8\",\"b6f3972d040447ad9044566b1bfb9bd4\",\"4dcb8d94d6cc42e9ad4245b3ced5d73a\",\"01a915c53cf34814ba59f798c46bff0a\",\"9c37e197eb0f4a79acbf86dfcfda9ee3\",\"a6dc6b641a1d4589baaf94c073b166c6\",\"66c92c089d6742d19a3ddd6ab093acce\",\"5efcb4f54e96445aab0bf6cd5c48e140\",\"64656ae50fcd435cabe838ba3fc74bf4\",\"a33c0c07eeb9433197cfc0012faaf86b\",\"d6388c264ec14c8591f2e0e8a4f508b5\",\"1f13df73c8734c6ea01dc84264d7a9a4\",\"a454a7ad6d214b46b2e22029e32713b3\",\"ab1afeb075db48dc852b38f7efb0b59a\",\"ed41ea29459b4a7297f99f56e5f5fac2\",\"5c12435c34304f78886c0dc85e812633\",\"9499cc5b0de94627addecd8765561db0\",\"8b78eb20237341758740e516ace41299\",\"d19283e264d44366acdaca17ebab70ad\",\"8b4541bc84794e399e16b0ac1f2ce5db\",\"d8ad3826d40b41c4934d896245794382\",\"b6773dc9fc6b40e2a89747ad88ac5f9d\",\"f2fcf88b1d1d4c8db80585600f373092\",\"8920704065a84c739381b3f6b8a71d33\",\"de37a9145f1f40928adf13224646523d\",\"c2baea7ddf4d4b2ebc038a7c412f99dd\",\"15a486a5a98f4011bda3a821389ccdc0\",\"ca36015e10544f55a1acbdbaf03017f0\",\"6eacd26fd23f4530baf98b65a9d8c166\",\"a9bdcf8f6140463787f41d9fb8370806\",\"10fa33db762e48048107084ce0df9688\",\"a7827923f5a14835abf070fe1a417ce6\",\"23772f2ec9e64c8ea329874661c0e72c\",\"d472cafc3561461399958630cb81c348\",\"ce089a58329e474babea74a657b6a8a8\",\"84b7eb51e38d42c3a09a2ab1f87a419b\",\"258166a5ad444e11ae291d47c29d8e95\",\"8dde20c91feb49488b45aac477379320\",\"883a09808ba74aceb03ec26cb525cb32\",\"9b9272daa2cc484b99ef22536f1e5ae9\",\"442953634e084c868c4c9be0e8530001\",\"dcd3611436f44099af77b29a9556363b\",\"b58a3ecb60904570a073ae3520867feb\",\"9d8c6c0800cf484f9a79172407fe7168\",\"5724373f8d6a4076a292be596efbdca8\",\"c7e9aab70d1541e49a55c1433898a725\",\"4a53092d22514d63bcd8e4a7906f6c6b\",\"b9dc22c807fc42d5bb5e084f191e408a\",\"7d46c9e5023543e2b82da0cf7f6bb0c4\"]}" id="wqwlOVH00TbB" outputId="1803fa26-e51e-4b9b-8f0d-5ac0d322f2e9"}
``` python
llm = Anthropic(temperature=0.0, model="claude-opus-4-1")
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")
```
:::
::: {.cell .code execution_count="5" id="aIkLOdRz0Unp"}
``` python
from llama_index.core import Settings
Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512
```
:::
::: {.cell .markdown id="JP-m752eKRWW"}
### Download Document
:::
:::: {.cell .code execution_count="6" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="MozCygK32Zy0" outputId="0f8d3446-3445-419c-a976-4405c0d93759"}
``` python
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/jerryjliu/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
```
::: {.output .stream .stdout}
--2024-03-08 07:04:27-- https://raw.githubusercontent.com/jerryjliu/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75042 (73K) [text/plain]
Saving to: ‘data/paul_graham/paul_graham_essay.txt’
data/paul_graham/pa 100%[===================>] 73.28K --.-KB/s in 0.002s
2024-03-08 07:04:27 (28.6 MB/s) - ‘data/paul_graham/paul_graham_essay.txt’ saved [75042/75042]
:::
::::
::: {.cell .markdown id="G4nEvK7CKTbh"}
### Load Document
:::
::: {.cell .code execution_count="7" id="n4AGgEvf0dpo"}
``` python
# load documents
from llama_index.core import SimpleDirectoryReader
documents = SimpleDirectoryReader("data/paul_graham").load_data()
```
:::
::: {.cell .markdown id="rAlVBGBOKYmX"}
### Create Indices and Query Engines. {#create-indices-and-query-engines}
:::
::: {.cell .code execution_count="8" id="yabW13VV2Vus"}
``` python
from llama_index.core import SummaryIndex, VectorStoreIndex
# Summary Index for summarization questions
summary_index = SummaryIndex.from_documents(documents)
# Vector Index for answering specific context questions
vector_index = VectorStoreIndex.from_documents(documents)
```
:::
::: {.cell .code execution_count="9" id="GQCnJnOl2i1M"}
``` python
# Summary Index Query Engine
summary_query_engine = summary_index.as_query_engine(
response_mode="tree_summarize",
use_async=True,
)
# Vector Index Query Engine
vector_query_engine = vector_index.as_query_engine()
```
:::
::: {.cell .markdown id="9gxEcFxLKeSN"}
### Create tools for summary and vector query engines. {#create-tools-for-summary-and-vector-query-engines}
:::
::: {.cell .code execution_count="10" id="qM0eVXy62ohZ"}
``` python
from llama_index.core.tools.query_engine import QueryEngineTool
# Summary Index tool
summary_tool = QueryEngineTool.from_defaults(
query_engine=summary_query_engine,
description="Useful for summarization questions related to Paul Graham eassy on What I Worked On.",
)
# Vector Index tool
vector_tool = QueryEngineTool.from_defaults(
query_engine=vector_query_engine,
description="Useful for retrieving specific context from Paul Graham essay on What I Worked On.",
)
```
:::
::: {.cell .markdown id="psgZak7mKkn3"}
### Create Router Query Engine
:::
::: {.cell .code execution_count="11" id="atzdlxci2rPk"}
``` python
from llama_index.core.query_engine.router_query_engine import RouterQueryEngine
from llama_index.core.selectors.llm_selectors import LLMSingleSelector
```
:::
::: {.cell .code execution_count="12" id="AX7laiwj2ut8"}
``` python
# Create Router Query Engine
query_engine = RouterQueryEngine(
selector=LLMSingleSelector.from_defaults(),
query_engine_tools=[
summary_tool,
vector_tool,
],
)
```
:::
::: {.cell .markdown id="r_nIEGuNKor4"}
### Test Queries
:::
:::: {.cell .code execution_count="13" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="RCeKGLxG2wZ-" outputId="86b24d46-31c2-4370-9323-95b2573c966a"}
``` python
response = query_engine.query("What is the summary of the document?")
```
::: {.output .stream .stdout}
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
Selecting query engine 0: The question is asking for a summary of the document. Choice 1 specifically mentions that it is useful for summarization questions related to Paul Graham's essay on What I Worked On, making it the most relevant choice for answering the given question..
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
:::
::::
:::: {.cell .code execution_count="14" colab="{\"base_uri\":\"https://localhost:8080/\",\"height\":128}" id="9e74o31q3kok" outputId="49db0722-b863-495f-b6b3-8067e09555d1"}
``` python
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))
```
::: {.output .display_data}
<p style="font-size:20px">The document is an autobiographical essay by Paul Graham, describing the major projects and events in his life from childhood through his 50s. Key points include:
- As a child, he was interested in programming and writing. He attended college intending to study philosophy but switched to AI.
- After grad school, he decided to pursue art and attended RISD and the Accademia in Florence. He supported himself doing freelance Lisp programming.
- In 1995, he and Robert Morris started Viaweb, one of the first web application companies, which was acquired by Yahoo in 1998. This made Graham wealthy.
- After leaving Yahoo, he returned to painting for a time, then started publishing essays online and working on a new Lisp dialect called Arc.
- In 2005, he co-founded Y Combinator, a new kind of startup investment firm, with Jessica Livingston, Robert Morris and Trevor Blackwell. He was very engaged in YC for many years as it pioneered a new model of startup funding.
- In 2013 he handed over the reins of YC to Sam Altman. After a period focused on painting, in 2015 he began developing a new Lisp dialect called Bel, which he worked on intensively for 4 years.
- The essay reflects on the winding path his career took, the accidental discoveries like the YC model, and the way his interests in art, writing and programming languages have intertwined over the decades. It emphasizes the value he found in pursuing ideas despite their unprestigious status.</p>
:::
::::
:::: {.cell .code execution_count="17" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="QnuVMeJt2yE2" outputId="64b49891-4c61-43c3-c622-2fa94c3f5c62"}
``` python
response = query_engine.query("What did Paul Graham do growing up?")
```
::: {.output .stream .stdout}
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
Selecting query engine 1: The question asks about specific details from Paul Graham's life, which would likely be found in the original essay. A summary of the essay may not include all the relevant details about what he did growing up..
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
:::
::::
:::: {.cell .code execution_count="18" colab="{\"base_uri\":\"https://localhost:8080/\",\"height\":128}" id="otNC-PzI20dM" outputId="d1d389e7-57d8-4eda-bca4-14b78d549b40"}
``` python
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))
```
::: {.output .display_data}
<p style="font-size:20px">According to the context provided, the two main things Paul Graham worked on outside of school before college were writing and programming. He wrote short stories, which he admits were awful, with hardly any plot and just characters with strong feelings.
In 9th grade, when he was 13 or 14, he started trying to write programs on an IBM 1401 computer that his school district used. The language was an early version of Fortran, and programs had to be typed on punch cards. However, he was puzzled by the 1401 and couldn't figure out what to really do with it, since the only input was data on punched cards which he didn't have. His clearest memory is learning that programs could fail to terminate when one of his didn't, which was a social faux pas on a shared machine. But everything changed for him once microcomputers became available.</p>
:::
::::
---
## SubQuestionQueryEngine
Source: https://github.com/anthropics/claude-cookbooks/blob/main/third_party/LlamaIndex/SubQuestion_Query_Engine.ipynb
Path: 11-RAG-Search/claude-cookbooks-third-party-llamaindex-subquestion-query-engine.md
# SubQuestionQueryEngine
Often, we encounter scenarios where our queries span across multiple
documents.
In this notebook, we delve into addressing complex queries that extend
over various documents by breaking them down into simpler sub-queries
and generate answers using the `SubQuestionQueryEngine`.
:::
::: {.cell .markdown id="jrGtIJogMsC5"}
### Installation
:::
::: {.cell .code id="s2vK4JUD4Yxg"}
``` python
!pip install llama-index
!pip install llama-index-llms-anthropic
!pip install llama-index-embeddings-huggingface
```
:::
::: {.cell .markdown id="BK17UTVoMuns"}
### Setup API Key
:::
::: {.cell .code execution_count="1" id="2XlJDsnB4v2_"}
``` python
import os
os.environ["ANTHROPIC_API_KEY"] = "YOUR Claude API KEY"
```
:::
::: {.cell .markdown id="toeP1CYRMxuf"}
### Setup LLM and Embedding model
We will use anthropic latest released `Claude-3 Opus` LLM.
:::
::: {.cell .code execution_count="2" id="E6xhHINq4ynN"}
``` python
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.anthropic import Anthropic
```
:::
::: {.cell .code execution_count="3" colab="{\"base_uri\":\"https://localhost:8080/\",\"height\":313,\"referenced_widgets\":[\"80fc006d4df8498a816e17cefc18ca2d\",\"e79bcee6519a4b6ab01b1bfa337cfe10\",\"66eef0ef7951416cbd6533d4845629cf\",\"d6d9e55994814729b3b68a79ba4198ce\",\"e460d00a45084a91a3cc0567bc8354e9\",\"879fa26aa4a04edeb5b79215067ce019\",\"e41c5f311d0e47e98f20502601dc87f4\",\"896de50f7e914df087f789315410addf\",\"76c0154ec5fe4be9b1fc9cb50c4ce7cb\",\"d7fe74b431364318b71c54f50f5062d2\",\"75f6b0b8bc8241deb83422857bb014c6\",\"6dbce7cfd7a6496dac0e0d75c3d3d89a\",\"a0b718321c544580b33665fc4d898c3f\",\"0c4f37adc854457a8e1e93e81f554909\",\"1ad42c23709242ae81e30144424342a4\",\"95414fde3b474d8987030ebad3123cf1\",\"fc448faa0d1349f3b2b99faf0d23df25\",\"16a40c3609114bff9d0574beebbd7f90\",\"f1a06c3d41ff4d6fa55ab10a8b968625\",\"bcaa0ae8f12c4c11b866711d130a5082\",\"3ac67761c43d4859a55b79e719ceb9eb\",\"5069cc3d6e444710a10464eb582d8667\",\"48c1eb1fef554a46a76573dc3175ddc9\",\"95305cdc84d74184ab385d51c18bc6b9\",\"c939510250db45f7a6ac46e1f9826c43\",\"b052069c77df4a8f9e53aae07bd5d354\",\"c752757087ba47bd9d66fc6937ec7e50\",\"eacd788bd8744cdc867799446cf96ece\",\"bd14648db88b47d79ec20a2b36eddd1a\",\"403c7c4189a647d59d101af4b9505bc4\",\"a3d5f40478804b3f8d74308777cbdb54\",\"8c5a99b093d94fe68dc95e002740ae45\",\"fb74a1b313964bda95083d682f5ee7ed\",\"db4a859d68274fad90dc1e734c603905\",\"042aa71a671c478b84e1f705f596cd8f\",\"545f291465ec4429b53047c8a553a947\",\"394a95ab8c2c4141ab87c602d163dad0\",\"7ee4bd7836774c1886019c336ba4eb0c\",\"5ffcec13f4354e0d941694fd31a6bb3a\",\"ef196e67ba4844fe84c40084931c6a45\",\"4db2d7d911b24198b2f2e0e627a4442b\",\"be1607bb5bc242e9a97e8e782b112ef2\",\"dc7d72d4cc2d4e1aaf4ebd1b9bdd7c34\",\"fd323e1105d744cb836e5388df45c8b7\",\"5b3061f6a3ba4b5aac88c9f5d93b51bc\",\"c117b7eebe0b46aa8999f5f507b83953\",\"30fbb2ead374431d8d595c2af9beb904\",\"ee1b883ed1f04a3c91a78be9003c6ce1\",\"d4e896d589e24ab39a7874756bbb53b7\",\"9566a05413924cfdb9f373b7512a9669\",\"ec4c3693cd51495082d9c1486d9ffecf\",\"0bdcd931790046ff85d806438979b905\",\"e44abbb2de714804a27906796acd141d\",\"48496204daf0451da5489bda293522db\",\"5938d2f27d564559861523e8aff1562c\",\"e9a8a9befc1e47a1ae664732d8393433\",\"24289de0731346fea8e800222afa93b9\",\"824931b749144703ac79dfe489e9ea05\",\"107683e48eed4188886fc1d78d61e240\",\"a8e9a6e25e3b48029a943af73e62dd3c\",\"51320c89d8dd41dd802e76e047d980a9\",\"5121931a2b614938b9f2b1b5a511e022\",\"013ede19019f48c693882cc9caf3ff51\",\"20bb8e32267c4b628c2daa337b119105\",\"4186402c66e446dd9e12545c6a19f30a\",\"400027c1605a4676833f6ea0810699ab\"]}" id="GIsJJBFo40Jq" outputId="ed4d7267-327a-4ead-a61f-801ae9a69481"}
``` python
llm = Anthropic(temperature=0.0, model="claude-opus-4-1")
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")
```
:::
::: {.cell .code execution_count="4" id="2Hqi93hb41nV"}
``` python
from llama_index.core import Settings
Settings.llm = llm
Settings.embed_model = embed_model
Settings.chunk_size = 512
```
:::
::: {.cell .markdown id="ZM0Ksh5QM1yc"}
### Setup logging
:::
::: {.cell .code execution_count="5" id="pY6CL0_e429M"}
``` python
# NOTE: This is ONLY necessary in jupyter notebook.
# Details: Jupyter runs an event-loop behind the scenes.
# This results in nested event-loops when we start an event-loop to make async queries.
# This is normally not allowed, we use nest_asyncio to allow it for convenience.
import nest_asyncio
nest_asyncio.apply()
import logging
import sys
# Set up the root logger
logger = logging.getLogger()
logger.setLevel(logging.INFO) # Set logger level to INFO
# Clear out any existing handlers
logger.handlers = []
# Set up the StreamHandler to output to sys.stdout (Colab's output)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO) # Set handler level to INFO
# Add the handler to the logger
logger.addHandler(handler)
from IPython.display import HTML, display
```
:::
::: {.cell .markdown id="MhFt3P-IM4ai"}
### Download Data
We will use Uber and Lyft 2021 10K SEC Filings
:::
:::: {.cell .code execution_count="6" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="ukHLfRJC44fG" outputId="2f5f2a1b-645d-4fa4-f6ab-b8f65373ba07"}
``` python
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/10k/uber_2021.pdf' -O './uber_2021.pdf'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/10k/lyft_2021.pdf' -O './lyft_2021.pdf'
```
::: {.output .stream .stdout}
--2024-03-08 07:07:32-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/10k/uber_2021.pdf
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1880483 (1.8M) [application/octet-stream]
Saving to: ‘./uber_2021.pdf’
./uber_2021.pdf 100%[===================>] 1.79M --.-KB/s in 0.02s
2024-03-08 07:07:32 (87.4 MB/s) - ‘./uber_2021.pdf’ saved [1880483/1880483]
--2024-03-08 07:07:33-- https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/10k/lyft_2021.pdf
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1440303 (1.4M) [application/octet-stream]
Saving to: ‘./lyft_2021.pdf’
./lyft_2021.pdf 100%[===================>] 1.37M --.-KB/s in 0.02s
2024-03-08 07:07:33 (74.9 MB/s) - ‘./lyft_2021.pdf’ saved [1440303/1440303]
:::
::::
::: {.cell .markdown id="ItM1L4OmM-Wb"}
### Load Data
:::
::: {.cell .code execution_count="7" id="pltY8vjU59mQ"}
``` python
from llama_index.core import SimpleDirectoryReader
lyft_docs = SimpleDirectoryReader(input_files=["lyft_2021.pdf"]).load_data()
uber_docs = SimpleDirectoryReader(input_files=["uber_2021.pdf"]).load_data()
```
:::
:::: {.cell .code execution_count="8" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="7dTY-clm5_k_" outputId="7ef4e12e-6348-476b-cab0-a3e38b2c5c98"}
``` python
print(f"Loaded lyft 10-K with {len(lyft_docs)} pages")
print(f"Loaded Uber 10-K with {len(uber_docs)} pages")
```
::: {.output .stream .stdout}
Loaded lyft 10-K with 238 pages
Loaded Uber 10-K with 307 pages
:::
::::
::: {.cell .markdown id="xSDF_QEsNBci"}
### Index Data
:::
::: {.cell .code execution_count="9" colab="{\"background_save\":true}" id="LRAZWegi6Dyi"}
``` python
from llama_index.core import VectorStoreIndex
lyft_index = VectorStoreIndex.from_documents(lyft_docs[:100])
uber_index = VectorStoreIndex.from_documents(uber_docs[:100])
```
:::
::: {.cell .markdown id="p6hwyc5DNGMk"}
### Create Query Engines
:::
::: {.cell .code execution_count="10" id="EHmUyRcC6F5f"}
``` python
lyft_engine = lyft_index.as_query_engine(similarity_top_k=5)
```
:::
::: {.cell .code execution_count="11" id="15hf7apf6HWF"}
``` python
uber_engine = uber_index.as_query_engine(similarity_top_k=5)
```
:::
::: {.cell .markdown id="ebyg3tfNNIzM"}
### Querying
:::
::::: {.cell .code execution_count="12" colab="{\"background_save\":true}" id="_6-FAxif6InL" outputId="4d1adf30-cc28-4f79-a906-aabf82a03e8d"}
``` python
response = await lyft_engine.aquery(
"What is the revenue of Lyft in 2021? Answer in millions with page reference"
)
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))
```
::: {.output .stream .stdout}
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
:::
::: {.output .display_data}
<p style="font-size:20px">According to the consolidated statements of operations on page 79, Lyft's total revenue for the year ended December 31, 2021 was $3,208.3 million.</p>
:::
:::::
::::: {.cell .code execution_count="13" colab="{\"background_save\":true}" id="2VSINSWB6SjI" outputId="9b5e0403-b613-464a-835a-e15689bc86b5"}
``` python
response = await uber_engine.aquery(
"What is the revenue of Uber in 2021? Answer in millions, with page reference"
)
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))
```
::: {.output .stream .stdout}
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
:::
::: {.output .display_data}
<p style="font-size:20px">According to the Consolidated Statements of Operations on page 77, Uber's revenue for the year ended December 31, 2021 was $17,455 million.</p>
:::
:::::
::: {.cell .markdown id="przIf6LINKuI"}
### Create Tools
:::
::: {.cell .code execution_count="14" id="kgK60jJq6VEz"}
``` python
from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata
query_engine_tools = [
QueryEngineTool(
query_engine=lyft_engine,
metadata=ToolMetadata(
name="lyft_10k", description="Provides information about Lyft financials for year 2021"
),
),
QueryEngineTool(
query_engine=uber_engine,
metadata=ToolMetadata(
name="uber_10k", description="Provides information about Uber financials for year 2021"
),
),
]
```
:::
::: {.cell .markdown id="kL1xN3SnNNr_"}
### Create `SubQuestionQueryEngine`
:::
::: {.cell .code execution_count="15" id="d-4zLgq8NTaR"}
``` python
sub_question_query_engine = SubQuestionQueryEngine.from_defaults(
query_engine_tools=query_engine_tools
)
```
:::
::: {.cell .markdown id="umqoF0qNNZL8"}
### Querying {#querying}
:::
:::: {.cell .code execution_count="17" colab="{\"base_uri\":\"https://localhost:8080/\",\"height\":409}" id="k5wLiAiD6Xsy" outputId="2a7b85df-0be9-4f9e-be45-3413d9838395"}
``` python
response = await sub_question_query_engine.aquery(
"Compare revenue growth of Uber and Lyft from 2020 to 2021"
)
```
::: {.output .stream .stdout}
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
Generated 4 sub questions.
[uber_10k] Q: What was Uber's revenue in 2020?
[uber_10k] Q: What was Uber's revenue in 2021?
[lyft_10k] Q: What was Lyft's revenue in 2020?
[lyft_10k] Q: What was Lyft's revenue in 2021?
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
[lyft_10k] A: According to Lyft's consolidated statements of operations data, Lyft's total revenue in 2020 was $2,364,681,000.
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
[uber_10k] A: According to Uber's consolidated statements of operations, Uber's revenue in 2021 was $17,455 million.
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
[uber_10k] A: According to Uber's consolidated statements of operations, Uber's revenue in 2020 was $11,139 million.
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
[lyft_10k] A: According to Lyft's consolidated statements of operations, Lyft's total revenue in 2021 was $3,208,323,000. This consisted of:
- Revenue from contracts with customers (under ASC 606) of $2,957,979,000
- Rental revenue (under ASC 842) of $250,344,000
So in total, Lyft generated revenue of $3,208,323,000 in the year ended December 31, 2021.
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
:::
::::
:::: {.cell .code execution_count="18" id="PgVULcF6I0bH"}
``` python
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))
```
::: {.output .display_data}
<p style="font-size:20px">Uber and Lyft both saw significant revenue growth from 2020 to 2021:
Uber's revenue increased from $11,139 million in 2020 to $17,455 million in 2021. This represents year-over-year growth of about 56.7%.
Lyft's total revenue grew from $2,364,681,000 in 2020 to $3,208,323,000 in 2021, an increase of approximately 35.7%.
So while Lyft started from a lower revenue base, Uber achieved higher revenue growth in percentage terms between 2020 and 2021 (56.7% vs 35.7%).
Both companies rebounded strongly in 2021 after seeing major impacts to their ride-sharing businesses in 2020 due to the COVID-19 pandemic. But Uber was able to grow its revenue more rapidly to reach $17.5 billion, compared to $3.2 billion for Lyft.</p>
:::
::::
:::: {.cell .code execution_count="19"}
``` python
response = await sub_question_query_engine.aquery("Compare the investments made by Uber and Lyft")
```
::: {.output .stream .stdout}
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
Generated 4 sub questions.
[uber_10k] Q: What investments did Uber make in 2021
[uber_10k] Q: What was the total amount invested by Uber in 2021
[lyft_10k] Q: What investments did Lyft make in 2021
[lyft_10k] Q: What was the total amount invested by Lyft in 2021
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
[uber_10k] A: Based on the context provided, in 2021 Uber invested:
- $2.3 billion in acquisition of businesses, net of cash acquired
- $1.1 billion in purchases of marketable securities
- $982 million in purchases of non-marketable equity securities
- $297 million in purchases of notes receivable
- $298 million in purchases of property and equipment
So in total, Uber invested approximately $5.0 billion in 2021 across business acquisitions, marketable and non-marketable securities, notes receivable, and property and equipment purchases.
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
[lyft_10k] A: Based on the context provided, in 2021 Lyft invested in marketable securities and term deposits:
- Lyft purchased marketable securities of $3.8 billion in 2021. The marketable securities consisted of investment grade available-for-sale debt securities.
- Lyft also invested in term deposits of $0.5 billion in 2021. The term deposits were at cost, which approximated fair value.
As of December 31, 2021, Lyft's investment portfolio had a weighted-average remaining maturity of less than one year. Lyft's investment policy is designed to minimize exposure to credit losses.
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
[uber_10k] A: Based on the provided context, in 2021 Uber made investments to:
- Increase the number of Drivers, consumers, merchants, shippers, and carriers using their platform through incentives, discounts, and promotions
- Expand within existing markets and into new markets
- Increase research and development expenses
- Expand marketing channels and operations
- Hire additional employees
- Add new products and offerings to their platform
The context indicates Uber expected to incur losses in the near term as a result of substantial increases in operating expenses from continuing to make these types of investments.
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
[lyft_10k] A: Based on the financial information provided in the context, the cash flow statement shows that Lyft had net cash provided by investing activities of $267.0 million for the year ended December 31, 2021. This primarily consisted of:
- Proceeds from sales and maturities of marketable securities of $3.8 billion
- Maturities of term deposits of $675.5 million
- Partially offset by purchases of marketable securities of $3.8 billion and term deposits of $0.5 billion
So while the total proceeds from sales/maturities was around $4.5 billion, Lyft reinvested most of that, with net new investments of approximately $267 million in 2021.
HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
:::
::::
:::: {.cell .code execution_count="20"}
``` python
display(HTML(f'<p style="font-size:20px">{response.response}</p>'))
```
::: {.output .display_data}
<p style="font-size:20px">In 2021, Uber and Lyft made different types of investments:
Uber invested a total of approximately $5.0 billion, which included:
- $2.3 billion in acquisition of businesses
- $1.1 billion in marketable securities
- $982 million in non-marketable equity securities
- $297 million in notes receivable
- $298 million in property and equipment
Lyft had net investments of around $267 million, primarily in:
- Marketable securities, with purchases of $3.8 billion, offset by sales and maturities of $3.8 billion
- Term deposits of $0.5 billion, offset by maturities of $675.5 million
So while Uber invested heavily in acquiring businesses, securities, notes, and property, Lyft focused its investments on shorter-term marketable securities and term deposits, with most proceeds being reinvested. Uber's investments were substantially larger in total dollar amount compared to Lyft's net investments in 2021.</p>
:::
::::
---
## How to Build A RAG System Using Claude 3 And MongoDB
Source: https://github.com/anthropics/claude-cookbooks/blob/main/third_party/MongoDB/rag_using_mongodb.ipynb
Path: 11-RAG-Search/claude-cookbooks-third-party-mongodb-rag-using-mongodb.md
# How to Build A RAG System Using Claude 3 And MongoDB
This tutorial implements a chatbot prompted to take on the role of a
Venture capital tech Analyst. The chatbot is a naive RAG system with a
collection of tech news articles acting as its knowledge source. This
notebook covers the following:
1. Follow a comprehensive tutorial on setting up your development
environment, from installing necessary libraries to configuring a
MongoDB database.
2. Learn efficient data handling methods, including creating vector
search indexes and preparing data for ingestion and query
processing.
3. Understand how to employ Claude 3 models within the RAG system for
generating precise responses based on contextual information
retrieved from the database.
You will need the following:
- Claude API Key
- VoyageAI API Key
- Hugging Face Access Token
:::
::: {.cell .markdown}
## Step 1: Library installation, data loading and preparation
Below are brief explanations of the tools and libraries utilised within
the implementation code:
- anthropic: The official Python library for Anthropic that enables
access to state-of-the-art language models. This library provides
access to the Claude 3 family models, which can understand text and
images.
- datasets: This library is part of the Hugging Face ecosystem. By
installing \'datasets\', we gain access to a number of pre-processed
and ready-to-use datasets, which are essential for training and
fine-tuning machine learning models or benchmarking their performance.
- pandas: This data science library provides robust data structures and
methods for data manipulation, processing, and analysis.
- voyageai: This is the official Python client library for accessing
VoyageAI\'s suite of embedding models.
- pymongo: PyMongo is a Python toolkit for MongoDB. It enables
interactions with a MongoDB database.
:::
::: {.cell .code id="zkePkAfZLQ_R"}
``` python
!pip install pymongo datasets pandas anthropic voyageai
```
:::
::: {.cell .markdown}
The code snippet below executes the following steps:
1. Import Necessary Libraries:
- `os` for interacting with the operating system,
- `requests` for making HTTP requests,
- `BytesIO` from the io module to handle bytes objects like files in
memory,
- `pandas` (as pd) for data manipulation and analysis, and
- `userdata` from google.colab enables access to environment variables
stored in Google Colab secrets.
1. Function Definition: The `download_and_combine_parquet_files`
function is defined with two parameters:
- `parquet_file_urls`: a list of URLs as strings, each pointing to a
Parquet file that contains a sub-collection of the tech-news-embedding
dataset.
- `hf_token` is a string representing a Hugging Face authorization
token. Access tokens can be created or copied from the [Hugging Face
platform](https://huggingface.co/docs/hub/en/security-tokens#:~:text=To%20create%20an%20access%20token,you%27re%20ready%20to%20go!)
1. Download and Read Parquet Files: The function iterates over each URL
in parquet_file_urls. For each URL, it:
- Makes a GET request using the requests.get method, passing the URL and
the headers for authorization.
- Checks if the response status code is 200 (OK), indicating the request
was successful.
- If successful, it reads the content of the response into a BytesIO
object (to handle it as a file in memory), then uses
pandas.read_parquet to read the Parquet file from this object into a
Pandas DataFrame.
- Appends the DataFrame to the list all_dataframes.
1. Combine DataFrames: After downloading and reading all Parquet files
into DataFrames, there's a check to ensure that `all_dataframes` is
not empty. If there are DataFrames to work with, then all DataFrames
are concatenated into a single DataFrame using pd.concat, with
ignore_index=True to reindex the new combined DataFrame. This
combined DataFrame is the overall process output in the
`download_and_combine_parquet_files` function.
:::
::: {.cell .code id="VoY9qBGJNo2F"}
``` python
from io import BytesIO
import pandas as pd
import requests
from google.colab import userdata
def download_and_combine_parquet_files(parquet_file_urls, hf_token):
"""
Downloads Parquet files from the provided URLs using the given Hugging Face token,
and returns a combined DataFrame.
Parameters:
- parquet_file_urls: List of strings, URLs to the Parquet files.
- hf_token: String, Hugging Face authorization token.
Returns:
- combined_df: A pandas DataFrame containing the combined data from all Parquet files.
"""
headers = {"Authorization": f"Bearer {hf_token}"}
all_dataframes = []
for parquet_file_url in parquet_file_urls:
response = requests.get(parquet_file_url, headers=headers, timeout=60)
if response.status_code == 200:
parquet_bytes = BytesIO(response.content)
df = pd.read_parquet(parquet_bytes)
all_dataframes.append(df)
else:
print(
f"Failed to download Parquet file from {parquet_file_url}: {response.status_code}"
)
if all_dataframes:
combined_df = pd.concat(all_dataframes, ignore_index=True)
return combined_df
else:
print("No dataframes to concatenate.")
return None
```
:::
::: {.cell .markdown}
Below is a list of the Parquet files required for this tutorial. The
complete list of all files is located
[here](https://huggingface.co/datasets/MongoDB/tech-news-embeddings/tree/refs%2Fconvert%2Fparquet/default/train).
Each Parquet file represents approximately 45,000 data points.
In the code snippet below, a subset of the tech-news-embeddings dataset
is grouped into a single DataFrame, which is then assigned to the
variable `combined_df`.
:::
::: {.cell .code execution_count="3" id="8sa8PrlAOawR"}
``` python
# Uncomment the links below to load more data
# For the full list of data visit: https://huggingface.co/datasets/MongoDB/tech-news-embeddings/tree/refs%2Fconvert%2Fparquet/default/train
parquet_files = [
"https://huggingface.co/api/datasets/AIatMongoDB/tech-news-embeddings/parquet/default/train/0000.parquet",
# "https://huggingface.co/api/datasets/AIatMongoDB/tech-news-embeddings/parquet/default/train/0001.parquet",
# "https://huggingface.co/api/datasets/AIatMongoDB/tech-news-embeddings/parquet/default/train/0002.parquet",
# "https://huggingface.co/api/datasets/AIatMongoDB/tech-news-embeddings/parquet/default/train/0003.parquet",
# "https://huggingface.co/api/datasets/AIatMongoDB/tech-news-embeddings/parquet/default/train/0004.parquet",
# "https://huggingface.co/api/datasets/AIatMongoDB/tech-news-embeddings/parquet/default/train/0005.parquet",
]
hf_token = userdata.get("HF_TOKEN")
combined_df = download_and_combine_parquet_files(parquet_files, hf_token)
```
:::
::: {.cell .markdown}
As a final phase in data preparation, the code snippet below shows the
step to remove the `_id` column from the grouped dataset, as it is
unnecessary for subsequent steps in this tutorial. Additionally, the
data within the embedding column for each data point is converted from a
numpy array to a Python list to prevent errors related to incompatible
data types during the data ingestion.
:::
::: {.cell .code execution_count="4" id="GEx4i1ehRliI"}
``` python
# Remove the _id coloum from the intital dataset
combined_df = combined_df.drop(columns=["_id"])
# Remove the initial embedding coloumn as we are going to create new embeddings with VoyageAI embedding model
combined_df = combined_df.drop(columns=["embedding"])
```
:::
::: {.cell .code colab="{\"base_uri\":\"https://localhost:8080/\",\"height\":293}" id="6UsKPi32OWX3" outputId="231e7041-7b4d-49f7-c7c5-1d81e7e4adea"}
``` python
combined_df.head()
```
:::
::: {.cell .code}
``` python
# Limiting the amount of document used to 500 for this demo due to the rate limit on VoyageAI API
# Read more on VoyageAI rate limits: https://docs.voyageai.com/docs/rate-limits
max_documents = 500
if len(combined_df) > max_documents:
combined_df = combined_df[:max_documents]
```
:::
::: {.cell .code}
``` python
import voyageai
vo = voyageai.Client(api_key=userdata.get("VOYAGE_API_KEY"))
def get_embedding(text: str) -> list[float]:
if not text.strip():
print("Attempted to get embedding for empty text.")
return []
embedding = vo.embed(text, model="voyage-large-2", input_type="document")
return embedding.embeddings[0]
combined_df["embedding"] = combined_df["description"].apply(get_embedding)
combined_df.head()
```
:::
::: {.cell .markdown id="GXv9MrqLSZw8"}
## Step 2: Database and collection creation
**To create a new MongoDB database, set up a database cluster:**
1. Register for a [free MongoDB Atlas
account](https://www.mongodb.com/cloud/atlas/register?utm_campaign=devrel&utm_source=community&utm_medium=cta&utm_content=Partner%20Cookbook&utm_term=richmond.alake),
or existing users, [sign into MongoDB
Atlas](https://account.mongodb.com/account/login?utm_campaign=devrel&utm_source=community&utm_medium=cta&utm_content=Partner%20Cookbook&utm_term=richmond.alake)
2. Select the "Database" option on the left-hand pane, which will
navigate to the Database Deployment page with a deployment
specification of any existing cluster. Create a new database cluster
by clicking on the \"+Create\" button.
3. For assistance with database cluster setup and obtaining the URI,
refer to our guide for setting up a MongoDB cluster and getting your
connection string. Note: Don't forget to whitelist the IP for the
Python host or 0.0.0.0/0 for any IP when creating proof of concepts.
4. After successfully creating and deploying the cluster, the cluster
becomes accessible on the 'Database Deployment' page.
5. Click on the "Connect" button of the cluster to view the option to
set up a connection to the cluster via various language drivers.
6. This tutorial only requires the cluster's URI(unique resource
identifier). Grab the URI and copy it into the Google Colabs Secrets
environment in a variable named MONGO_URI or place it in a .env file
or equivalent.
Once you have created a cluster, navigate to the cluster page and create
a database and collection within the MongoDB Atlas cluster by clicking +
Create Database. The database will be named `tech_news`, and the
collection will be named `hacker_noon_tech_news`.
:::
::: {.cell .markdown}
## Step 3: Vector search index creation
By this point, you have created a cluster, database and collection.
The steps in this section are crucial to ensure that a vector search can
be conducted using the queries entered into the chatbot and searched
against the records within the hacker_noon_tech_news collection. The
objective of this step is to create a vector search index. To achieve
this, refer to the official [vector search index creation
guide](https://www.mongodb.com/docs/atlas/atlas-vector-search/create-index/).
In the creation of a vector search index using the JSON editor on
MongoDB Atlas, ensure your vector search index is named vector_index and
the vector search index definition is as follows:
{
"fields": [{
"numDimensions": 1536,
"path": "embedding",
"similarity": "cosine",
"type": "vector"
}]
}
:::
::: {.cell .markdown}
## Step 4: Data ingestion
To ingest data into the MongoDB database created in previous steps. The
following operations have to be carried out:
- Connect to the database and collection
- Clear out the collection of any existing records
- Convert the Pandas DataFrame of the dataset into dictionaries before
ingestion
- Ingest dictionaries into MongoDB using a batch operation
This tutorial requires the cluster\'s URI (unique resource identifier).
Grab the URI and copy it into the Google Colab Secrets environment in a
variable named MONGO_URI, or place it in a .env file or equivalent.
:::
:::: {.cell .code execution_count="6" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="Yv0dMtWnQhDy" outputId="5bc5276b-46b3-4bc3-f5e4-19802ff910c3"}
``` python
import pymongo
from google.colab import userdata
def get_mongo_client(mongo_uri):
"""Establish connection to the MongoDB."""
try:
client = pymongo.MongoClient(mongo_uri)
print("Connection to MongoDB successful")
return client
except pymongo.errors.ConnectionFailure as e:
print(f"Connection failed: {e}")
return None
mongo_uri = userdata.get("MONGO_URI")
if not mongo_uri:
print("MONGO_URI not set in environment variables")
mongo_client = get_mongo_client(mongo_uri)
DB_NAME = "tech_news"
COLLECTION_NAME = "hacker_noon_tech_news"
db = mongo_client[DB_NAME]
collection = db[COLLECTION_NAME]
```
::: {.output .stream .stdout}
Connection to MongoDB successful
:::
::::
:::: {.cell .code execution_count="7" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="4NowJ2mlQrfT" outputId="31bfa7ac-dc62-421e-92c5-f644e03d7c1f"}
``` python
# To ensure we are working with a fresh collection
# delete any existing records in the collection
collection.delete_many({})
```
::: {.output .execute_result execution_count="7"}
DeleteResult({'n': 228012, 'electionId': ObjectId('7fffffff000000000000000e'), 'opTime': {'ts': Timestamp(1709660559, 7341), 't': 14}, 'ok': 1.0, '$clusterTime': {'clusterTime': Timestamp(1709660559, 7341), 'signature': {'hash': b'jT\xf1\xb4\xa9\xd3\xe3suu\x03`\x15(}\x8f\x00\x9f\xe9\x8a', 'keyId': 7320226449804230661}}, 'operationTime': Timestamp(1709660559, 7341)}, acknowledged=True)
:::
::::
::: {.cell .code id="tdFsJXXRjEZe"}
``` python
# Data Ingestion
combined_df_json = combined_df.to_dict(orient="records")
collection.insert_many(combined_df_json)
```
:::
::: {.cell .markdown}
## Step 5: Vector Search
This section showcases the creation of a vector search custom function
that accepts a user query, which corresponds to entries to the chatbot.
The function also takes a second parameter, `collection`, which points
to the database collection containing records against which the vector
search operation should be conducted.
The `vector_search` function produces a vector search result derived
from a series of operations outlined in a MongoDB aggregation pipeline.
This pipeline includes the `$vectorSearch` and `$project` stages and
performs queries based on the vector embeddings of user queries. It then
formats the results, omitting any record attributes unnecessary for
subsequent processes.
The code snippet below conducts the following operations to allow
semantic search for movies:
1. Define the `vector_search` function that takes a user\'s query
string and a MongoDB collection as inputs and returns a list of
documents that match the query based on vector similarity search.
2. Generate an embedding for the user\'s query by calling the
previously defined function, `get_embedding`, which converts the
query string into a vector representation.
3. Construct a pipeline for MongoDB\'s aggregate function,
incorporating two main stages: `$vectorSearch` and `$project`.
4. The `$vectorSearch` stage performs the actual vector search. The
index field specifies the vector index to utilise for the vector
search, and this should correspond to the name entered in the vector
search index definition in previous steps. The queryVector field
takes the embedding representation of the use query. The path field
corresponds to the document field containing the embeddings. The
`numCandidates` specifies the number of candidate documents to
consider and the limit on the number of results to return.
5. The \$project stage formats the results to excludes the \_id the
`embedding` field.
6. The aggregate executes the defined pipeline to obtain the vector
search results. The final operation converts the returned cursor
from the database into a list.
:::
::: {.cell .code execution_count="9" id="QsSBrAu4nSJC"}
``` python
def vector_search(user_query, collection):
"""
Perform a vector search in the MongoDB collection based on the user query.
Args:
user_query (str): The user's query string.
collection (MongoCollection): The MongoDB collection to search.
Returns:
list: A list of matching documents.
"""
# Generate embedding for the user query
query_embedding = get_embedding(user_query)
if query_embedding is None:
return "Invalid query or embedding generation failed."
# Define the vector search pipeline
pipeline = [
{
"$vectorSearch": {
"index": "vector_index",
"queryVector": query_embedding,
"path": "embedding",
"numCandidates": 150, # Number of candidate matches to consider
"limit": 5, # Return top 5 matches
}
},
{
"$project": {
"_id": 0, # Exclude the _id field
"embedding": 0, # Exclude the embedding field
"score": {
"$meta": "vectorSearchScore" # Include the search score
},
}
},
]
# Execute the search
results = collection.aggregate(pipeline)
return list(results)
```
:::
::: {.cell .markdown}
## Step 6: Handling user queries with Claude 3 models
The final section of the tutorial outlines the sequence of operations
performed as follows:
- Accept a user query in the form of a string.
- Utilize the VoyageAI embedding model to generate embeddings for the
user query.
- Load the Anthropic Claude 3, specifically the 'claude-opus-4-1' model,
to serve as the base model for the RAG system.
- Execute a vector search using the embeddings of the user query to
fetch relevant information from the knowledge base, which provides
additional context for the base model.
- Submit both the user query and the gathered additional information to
the base model to generate a response.
An important note is that the dimensions of the user query embedding
match the dimensions set in the vector search index definition on
MongoDB Atlas.
:::
::: {.cell .markdown}
The next step in this section is to import the anthropic library and
load the client to access the anthropic's methods for handling messages
and accessing Claude models. Ensure you obtain an Claude API key located
within the settings page on the [official Anthropic
website](https://console.anthropic.com/settings/keys).
:::
::: {.cell .code execution_count="11" id="wADfSOFOnw0s"}
``` python
import anthropic
client = anthropic.Client(api_key=userdata.get("ANTHROPIC_API_KEY"))
```
:::
::: {.cell .markdown}
Below is a more detailed description of the operations in the code
snippet below:
1. Vector Search Execution: The function begins by calling
`vector_search` with the user\'s query and a specified collection as
arguments. This performs a search within the collection, leveraging
vector embeddings to find relevant information related to the query.
2. Compile Search Results: `search_result` is initialized as an empty
string to aggregate information from the search. The search results
are compiled by iterating over the results returned by the
`vector_search` function, formates each item\'s details (title,
company name, URL, publication date, article URL, and description)
into a human-readable string, appending this information to
search_result with a newline character \\n at the end of each entry.
3. Generate Response Using Anthropic Client: The function then
constructs a request to the Claude API (through a client object,
presumably an instance of the anthropic. Client class created
earlier). It specifies:
- The model to use (\"claude-opus-4-1\") indicates a specific version of
the Claude 3 model.
- The maximum token limit for the generated response (max_tokens=1024).
- A system description guides the model to behave as a \"Venture Capital
Tech Analyst\" with access to tech company articles and information,
using this context to advise.
- The actual message for the model to process combines the user query
with the aggregated search results as context.
1. Return the Generated Response and Search Results: It extracts and
returns the response text from the first item in the response\'s
content, alongside the compiled search results.
:::
::: {.cell .code execution_count="12" id="JrG8Ho50nhVR"}
``` python
def handle_user_query(query, collection):
get_knowledge = vector_search(query, collection)
search_result = ""
for result in get_knowledge:
search_result += (
f"Title: {result.get('title', 'N/A')}, "
f"Company Name: {result.get('companyName', 'N/A')}, "
f"Company URL: {result.get('companyUrl', 'N/A')}, "
f"Date Published: {result.get('published_at', 'N/A')}, "
f"Article URL: {result.get('url', 'N/A')}, "
f"Description: {result.get('description', 'N/A')}, \n"
)
response = client.messages.create(
model="claude-opus-4-1",
max_tokens=1024,
system="You are Venture Captital Tech Analyst with access to some tech company articles and information. You use the information you are given to provide advice.",
messages=[
{
"role": "user",
"content": "Answer this user query: "
+ query
+ " with the following context: "
+ search_result,
}
],
)
return (response.content[0].text), search_result
```
:::
::: {.cell .markdown}
The final step in this tutorial is to initialize the query, pass it into
the `handle_user_query` function and print the response returned.
:::
:::: {.cell .code execution_count="13" colab="{\"base_uri\":\"https://localhost:8080/\"}" id="_3OcaBbmoY2H" outputId="06c231ef-cbaa-48cd-e226-9d1c8450b737"}
``` python
# Conduct query with retrieval of sources
query = "Give me the best tech stock to invest in and tell me why"
response, source_information = handle_user_query(query, collection)
print(f"Response: {response}")
print(f"\\nSource Information: \\n{source_information}")
```
::: {.output .stream .stdout}
Response: Based on the information provided in the article titles and descriptions, Alibaba Group Holding Limited appears to be a top technology stock pick for 2023 according to renowned investor Ray Dalio. The article "Top 10 Technology Stocks to Buy in 2023 According to Ray Dalio" suggests that Alibaba is one of Dalio's favored tech investments for the year.
As a venture capital tech analyst, I would recommend considering an investment in Alibaba for the following reasons:
1. Endorsement from a respected investor: Ray Dalio, known for his successful investment strategies, has included Alibaba in his top 10 technology stock picks for 2023. His backing lends credibility to the investment potential of the company.
2. Strong market position: Alibaba is a leading e-commerce company in China with a significant market share. It has a diversified business model spanning e-commerce, cloud computing, digital media, and entertainment.
3. Growth potential: With China's large and growing middle class, Alibaba is well-positioned to benefit from increasing consumer spending and the shift towards online shopping.
However, it's essential to consider the following points as well:
1. Regulatory risks: Chinese tech companies, including Alibaba, have faced increased regulatory scrutiny in recent times. Changes in government policies could impact the company's growth and profitability.
2. Competition: While Alibaba is a dominant player, it faces competition from other tech giants like Tencent and JD.com in various business segments.
3. Geopolitical tensions: Ongoing tensions between the U.S. and China could lead to market volatility and impact investor sentiment towards Chinese stocks.
As with any investment, it's crucial to conduct thorough research, consider your risk tolerance, and diversify your portfolio. Keep in mind that the information provided here is based on limited data points, and stock prices can be influenced by various factors beyond the scope of this context.
Source Information: \nTitle: Top 10 Technology Stocks to Buy in 2023 According to Ray Dalio, Company Name: ALIBABA GROUP HOLDING LIMITED, Company URL: https://hackernoon.com/company/alibabagroupholdinglimited, Date Published: 2023-04-21 11:58:00, Article URL: https://uk.finance.yahoo.com/news/top-10-technology-stocks-buy-155830366.html, Description: In this article we discuss the top 10 technology stocks to buy in 2023 according to Ray Dalio. If you want to skip our detailed analysis of Dalio’s investment philosophy and portfolio construction,
Title: 3 Tech Stocks I Love Right Now, Company Name: 10Clouds, Company URL: https://hackernoon.com/company/10clouds, Date Published: 2023-04-02 11:30:00, Article URL: https://www.msn.com/en-xl/money/other/3-tech-stocks-i-love-right-now/ar-AA19n9Ht, Description: These are tech giants but they''re also great investments.,
Title: 3 Millionaire-Maker Hydrogen Stocks to Buy Before the Window Closes, Company Name: Air Products & Chemicals, Company URL: https://hackernoon.com/company/airproductschemicals, Date Published: 2023-07-28 12:18:00, Article URL: https://www.msn.com/en-us/money/topstocks/3-millionaire-maker-hydrogen-stocks-to-buy-before-the-window-closes/ar-AA1etN8O, Description: These are the best hydrogen stocks to buy with multibagger returns potential.More From InvestorPlace Buy This $5 Stock BEFORE This Apple Project Goes Live Wall Street Titan: Here’s My #1 Stock for 2023 The $1 Investment You MUST Take Advantage of Right Now It doesn’t matter if you have $500 or $5 million.,
Title: Why it may be time to sell the pop in tech stocks: BlackRock, Company Name: BlackRock, Company URL: https://hackernoon.com/company/blackrock, Date Published: 2023-02-13 19:06:00, Article URL: https://news.yahoo.com/why-it-may-be-time-to-sell-the-pop-in-tech-stocks-blackrock-190606866.html, Description: Household tech names like Apple Meta and Netflix have soared so far in 2023 but one strategist says the gains aren''t likely to last. It''s time to take profits on tech stocks — the early sector winner of 2023 — as the Federal Reserve may soon dash hopes for a pivot on interest rates,
Title: The harsh reality for investors eyeing tech stocks in 2023: Morning Brief, Company Name: 10Clouds, Company URL: https://hackernoon.com/company/10clouds, Date Published: 2023-01-02 11:18:00, Article URL: https://news.yahoo.com/the-harsh-reality-for-investors-eyeing-tech-stocks-in-2023-morning-brief-111854804.html, Description: Curious on how to buy battered tech stocks? Here''s a quick tip. More on that and what else to watch in business on Monday.,
:::
::::
---
## Retrieval-Augmented Generation using Pinecone
Source: https://github.com/anthropics/claude-cookbooks/blob/main/third_party/Pinecone/rag_using_pinecone.ipynb
Path: 11-RAG-Search/claude-cookbooks-third-party-pinecone-rag-using-pinecone.md
# Retrieval-Augmented Generation using Pinecone
This notebook demonstrates how to connect Claude with the data in your
Pinecone vector database through a technique called retrieval-augmented
generation (RAG). We will cover the following steps:
1. Embedding a dataset using Voyage AI\'s embedding model
2. Uploading the embeddings to a Pinecone index
3. Retrieving information from the vector database
4. Using Claude to answer questions with information from the database
:::
::: {.cell .markdown}
## Setup
First, let\'s install the necessary libraries and set the API keys we
will need to use in this notebook. We will need to get a [Claude API
key](https://docs.claude.com/claude/reference/getting-started-with-the-api),
a free [Pinecone API key](https://docs.pinecone.io/docs/quickstart), and
a free [Voyage AI API key](https://docs.voyageai.com/install/).
:::
::: {.cell .code}
``` python
%pip install anthropic datasets pinecone-client voyageai
```
:::
::: {.cell .code}
``` python
# Insert your API keys here
ANTHROPIC_API_KEY = "<YOUR_ANTHROPIC_API_KEY>"
PINECONE_API_KEY = "<YOUR_PINECONE_API_KEY>"
VOYAGE_API_KEY = "<YOUR_VOYAGE_API_KEY>"
```
:::
::: {.cell .markdown}
## Download the dataset
Now let\'s download the Amazon products dataset which has over 10k
Amazon product descriptions and load it into a DataFrame.
:::
::: {.cell .code}
``` python
import pandas as pd
# Download the JSONL file
!wget https://www-cdn.anthropic.com/48affa556a5af1de657d426bcc1506cdf7e2f68e/amazon-products.jsonl
data = []
with open("amazon-products.jsonl") as file:
for line in file:
try:
data.append(eval(line)) # noqa: S307
except (SyntaxError, ValueError):
# Skip malformed lines in the dataset
pass
df = pd.DataFrame(data)
display(df.head())
len(df)
```
:::
::: {.cell .markdown}
## Vector Database
To create our vector database, we first need a free API key from
Pinecone. Once we have the key, we can initialize the database as
follows:
:::
::: {.cell .code}
``` python
from pinecone import Pinecone
pc = Pinecone(api_key=PINECONE_API_KEY)
```
:::
::: {.cell .markdown}
Next, we set up our index specification, which allows us to define the
cloud provider and region where we want to deploy our index. You can
find a list of all available providers and regions
[here](https://www.pinecone.io/docs/data-types/metadata/).
:::
::: {.cell .code}
``` python
from pinecone import ServerlessSpec
spec = ServerlessSpec(cloud="aws", region="us-west-2")
```
:::
::: {.cell .markdown}
Then, we initialize the index. We will be using Voyage\'s \"voyage-2\"
model for creating the embeddings, so we set the dimension to 1024.
:::
::: {.cell .code}
``` python
import time
index_name = "amazon-products"
existing_indexes = [index_info["name"] for index_info in pc.list_indexes()]
# check if index already exists (it shouldn't if this is first time)
if index_name not in existing_indexes:
# if does not exist, create index
pc.create_index(
index_name,
dimension=1024, # dimensionality of voyage-2 embeddings
metric="dotproduct",
spec=spec,
)
# wait for index to be initialized
while not pc.describe_index(index_name).status["ready"]:
time.sleep(1)
# connect to index
index = pc.Index(index_name)
time.sleep(1)
# view index stats
index.describe_index_stats()
```
:::
::: {.cell .markdown}
We should see that the new Pinecone index has a total_vector_count of 0,
as we haven\'t added any vectors yet.
:::
::: {.cell .markdown}
## Embeddings
To get started with Voyage\'s embeddings, go
[here](https://www.voyageai.com) to get an API key.
Now let\'s set up our Voyage client and demonstrate how to create an
embedding using the `embed` method. To learn more about using Voyage
embeddings with Claude, see [this
notebook](https://github.com/anthropics/anthropic-cookbook/blob/main/third_party/VoyageAI/how_to_create_embeddings.md).
:::
::: {.cell .code}
``` python
import voyageai
vo = voyageai.Client(api_key=VOYAGE_API_KEY)
texts = ["Sample text 1", "Sample text 2"]
result = vo.embed(texts, model="voyage-2", input_type="document")
print(result.embeddings[0])
print(result.embeddings[1])
```
:::
::: {.cell .markdown}
## Uploading data to the Pinecone index
With our embedding model set up, we can now take our product
descriptions, embed them, and upload the embeddings to the Pinecone
index.
:::
::: {.cell .code}
``` python
from time import sleep
from tqdm.auto import tqdm
descriptions = df["text"].tolist()
batch_size = 100 # how many embeddings we create and insert at once
for i in tqdm(range(0, len(descriptions), batch_size)):
# find end of batch
i_end = min(len(descriptions), i + batch_size)
descriptions_batch = descriptions[i:i_end]
# create embeddings (try-except added to avoid RateLimitError. Voyage currently allows 300/requests per minute.)
done = False
while not done:
try:
res = vo.embed(descriptions_batch, model="voyage-2", input_type="document")
done = True
except Exception:
sleep(5)
embeds = [record for record in res.embeddings]
# create unique IDs for each text
ids_batch = [f"description_{idx}" for idx in range(i, i_end)]
# Create metadata dictionaries for each text
metadata_batch = [{"description": description} for description in descriptions_batch]
to_upsert = list(zip(ids_batch, embeds, metadata_batch, strict=False))
# upsert to Pinecone
index.upsert(vectors=to_upsert)
```
:::
::: {.cell .markdown}
## Making queries
With our index populated, we can start making queries to get results. We
can take a natural language question, embed it, and query it against the
index to return semantically similar product descriptions.
:::
:::: {.cell .code execution_count="49"}
``` python
USER_QUESTION = (
"I want to get my daughter more interested in science. What kind of gifts should I get her?"
)
question_embed = vo.embed([USER_QUESTION], model="voyage-2", input_type="query")
results = index.query(vector=question_embed.embeddings, top_k=5, include_metadata=True)
results
```
::: {.output .execute_result execution_count="49"}
{'matches': [{'id': 'description_1771',
'metadata': {'description': 'Product Name: Scientific Explorer '
'My First Science Kids Science '
'Experiment Kit\n'
'\n'
'About Product: Experiments to spark '
'creativity and curiosity | Grow '
'watery crystals, create a rainbow '
'in a plate, explore the science of '
'color and more | Represents STEM '
'(Science, Technology, Engineering, '
'Math) principles – open ended toys '
'to construct, engineer, explorer '
'and experiment | Includes cross '
'linked polyacrylamide, 3 color '
'tablets, 3 mixing cups, 3 test '
'tubes, caps and stand, pipette, '
'mixing tray, magnifier and '
'instructions | Recommended for '
'children 4 years of age and older '
'with adult supervision\n'
'\n'
'Categories: Toys & Games | Learning '
'& Education | Science Kits & Toys'},
'score': 0.772703767,
'values': []},
{'id': 'description_3133',
'metadata': {'description': 'Product Name: Super Science Magnet '
'Kit.\n'
'\n'
'About Product: \n'
'\n'
'Categories: Toys & Games | Learning '
'& Education | Science Kits & Toys'},
'score': 0.765997052,
'values': []},
{'id': 'description_1792',
'metadata': {'description': 'Product Name: BRIGHT Atom Model - '
'Student\n'
'\n'
'About Product: \n'
'\n'
'Categories: Toys & Games | Learning '
'& Education | Science Kits & Toys'},
'score': 0.765654,
'values': []},
{'id': 'description_1787',
'metadata': {'description': 'Product Name: Thames & Kosmos '
'Biology Genetics and DNA\n'
'\n'
'About Product: Learn the basics of '
'genetics and DNA. | Assemble a '
'model to see the elegant '
'double-stranded Helical structure '
"of DNA. | A parents' Choice Gold "
'award winner | 20 experiments in '
'the 48 page full color experiment '
'manual and learning guide\n'
'\n'
'Categories: Toys & Games | Learning '
'& Education | Science Kits & Toys'},
'score': 0.765174091,
'values': []},
{'id': 'description_120',
'metadata': {'description': 'Product Name: Educational Insights '
"Nancy B's Science Club Binoculars "
'and Wildlife Activity Journal\n'
'\n'
'About Product: From bird search and '
'ecosystem challenges to creative '
'writing and drawing exercises, this '
'set is perfect for the nature lover '
'in your life! | Includes 4x '
'magnification binoculars and '
'22-page activity journal packed '
'with scientific activities! | '
'Binoculars are lightweight, yet '
'durable. | Supports STEM learning, '
'providing hands-on experience with '
'a key scientific tool. | Great '
'introductory tool for young '
'naturalists on-the-go! | Part of '
"the Nancy B's Science Club line, "
'designed to encourage scientific '
'confidence. | Winner of the '
"Parents' Choice Recommended Award. "
'| Scientific experience designed '
'specifically for kids ages 8-11.\n'
'\n'
'Categories: Electronics | Camera & '
'Photo | Binoculars & Scopes | '
'Binoculars'},
'score': 0.765075564,
'values': []}],
'namespace': '',
'usage': {'read_units': 6}}
:::
::::
::: {.cell .markdown}
## Optimizing search
These results are good, but we can optimize them even further. Using
Claude, we can take the user\'s question and generate search keywords
from it. This allows us to perform a wide, diverse search over the index
to get more relevant product descriptions.
:::
::: {.cell .code}
``` python
import anthropic
client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY)
def get_completion(prompt):
completion = client.completions.create(
model="claude-2.1",
prompt=prompt,
max_tokens_to_sample=1024,
)
return completion.completion
```
:::
::: {.cell .code}
``` python
def create_keyword_prompt(question):
return f"""\n\nHuman: Given a question, generate a list of 5 very diverse search keywords that can be used to search for products on Amazon.
The question is: {question}
Output your keywords as a JSON that has one property "keywords" that is a list of strings. Only output valid JSON.\n\nAssistant:{{"""
```
:::
::: {.cell .markdown}
With our Anthropic client setup and our prompt created, we can now begin
to generate keywords from the question. We will output the keywords in a
JSON object so we can easily parse them from Claude\'s output.
:::
::: {.cell .code}
``` python
keyword_json = "{" + get_completion(create_keyword_prompt(USER_QUESTION))
print(keyword_json)
```
:::
::: {.cell .code}
``` python
import json
# Extract the keywords from the JSON
data = json.loads(keyword_json)
keywords_list = data["keywords"]
print(keywords_list)
```
:::
::: {.cell .markdown}
Now with our keywords in a list, let\'s embed each one, query it against
the index, and return the top 3 most relevant product descriptions.
:::
::: {.cell .code}
``` python
results_list = []
for keyword in keywords_list:
# get the embeddings for the keywords
query_embed = vo.embed([keyword], model="voyage-2", input_type="query")
# search for the embeddings in the Pinecone index
search_results = index.query(vector=query_embed.embeddings, top_k=3, include_metadata=True)
# append the search results to the list
for search_result in search_results.matches:
results_list.append(search_result["metadata"]["description"])
print(len(results_list))
```
:::
::: {.cell .markdown}
## Answering with Claude
Now that we have a list of product descriptions, let\'s format them into
a search template Claude has been trained with and pass the formatted
descriptions into another prompt.
:::
::: {.cell .code execution_count="47"}
``` python
# Formatting search results
def format_results(extracted: list[str]) -> str:
result = "\n".join(
[
f'<item index="{i + 1}">\n<page_content>\n{r}\n</page_content>\n</item>'
for i, r in enumerate(extracted)
]
)
return f"\n<search_results>\n{result}\n</search_results>"
def create_answer_prompt(results_list, question):
return f"""\n\nHuman: {format_results(results_list)} Using the search results provided within the <search_results></search_results> tags, please answer the following question <question>{question}</question>. Do not reference the search results in your answer.\n\nAssistant:"""
```
:::
::: {.cell .markdown}
Finally, let\'s ask the original user\'s question and get our answer
from Claude.
:::
:::: {.cell .code execution_count="50"}
``` python
answer = get_completion(create_answer_prompt(results_list, USER_QUESTION))
print(answer)
```
::: {.output .stream .stdout}
To get your daughter more interested in science, I would recommend getting her an age-appropriate science kit or set that allows for hands-on exploration and experimentation. For example, for a younger child you could try a beginner chemistry set, magnet set, or crystal growing kit. For an older child, look for kits that tackle more advanced scientific principles like physics, engineering, robotics, etc. The key is choosing something that sparks her natural curiosity and lets her actively investigate concepts through activities, observations, and discovery. Supplement the kits with science books, museum visits, documentaries, and conversations about science she encounters in everyday life. Making science fun and engaging is crucial for building her interest.
:::
::::
---
## Embeddings
Source: https://github.com/anthropics/claude-cookbooks/blob/main/third_party/VoyageAI/how_to_create_embeddings.md
Path: 11-RAG-Search/claude-cookbooks-third-party-voyageai-how-to-create-embeddings.md
# Embeddings
Text embeddings are numerical representations of text strings, represented as a vector of floating point numbers. We can use the distance between two text embeddings (popularly cosine similarity) to measure how related two pieces of text are to one another, with smaller distances predicting higher relatedness.
Comparing the similarity of strings, or clustering strings by their distance from one another, allows for a wide variety of applications including **search** (popular in RAG architectures), **recommendations**, and **anomaly detection**.
## How to get embeddings with Anthropic
While Anthropic does not offer its own embedding model, we have partnered with [Voyage AI](https://www.voyageai.com/?ref=anthropic) as our preferred provider for text embeddings. Voyage makes [state of the art](https://blog.voyageai.com/2023/10/29/voyage-embeddings/?ref=anthropic) embedding models, and even offers models customized for specific industry domains such as finance and healthcare, and models that can be fine-tuned for your company.
To access Voyage embeddings, please first sign up on [Voyage AI’s website](https://dash.voyageai.com/?ref=anthropic), obtain an API key, and set the API key as an environment variable for convenience:
```bash
export VOYAGE_API_KEY="<your secret key>"
```
You can obtain the embeddings either using the official [`voyageai` Python package](https://github.com/voyage-ai/voyageai-python) or HTTP requests, as described below.
### Voyage Python Package
The `voyageai` package can be installed using the following command:
```bash
pip install -U voyageai
```
Then, you can create a client object and start using it to embed your texts:
```python
import voyageai
vo = voyageai.Client()
# This will automatically use the environment variable VOYAGE_API_KEY.
# Alternatively, you can use vo = voyageai.Client(api_key="<your secret key>")
texts = ["Sample text 1", "Sample text 2"]
result = vo.embed(texts, model="voyage-2", input_type="document")
print(result.embeddings[0])
print(result.embeddings[1])
```
`result.embeddings` will be a list of two embedding vectors, each containing 1024 floating-point numbers. After running the above code, the two embeddings will be printed on the screen:
```
[0.02012746, 0.01957859, ...] # embedding for "Sample text 1"
[0.01429677, 0.03077182, ...] # embedding for "Sample text 2"
```
When creating the embeddings, you may specify a few other arguments to the `embed()` function. Here is the specification:
> `voyageai.Client.embed(texts : List[str], model : str = "voyage-2", input_type : Optional[str] = None, truncation : Optional[bool] = None)`
- **texts** (List[str]) - A list of texts as a list of strings, such as `["I like cats", "I also like dogs"]`. Currently, the maximum length of the list is 128, and total number of tokens in the list is at most 320K for `voyage-2` and 120K for `voyage-code-2`.
- **model** (str) - Name of the model. Recommended options: `voyage-2` (default), `voyage-code-2`.
- **input_type** (str, optional, defaults to `None`) - Type of the input text. Defalut to `None`. Other options: `query`, `document`.
- When the input_type is set to `None`, and the input text will be directly encoded by our embedding model. Alternatively, when the inputs are documents or queries, the users can specify input_type to be `query` or `document`, respectively. In such cases, Voyage will prepend a special prompt to input text and send the extended inputs to the embedding model.
- For retrieval/search use cases, we recommend specifying this argument when encoding queries or documents to enhance retrieval quality. Embeddings generated with and without the input_type argument are compatible.
- **truncation** (bool, optional, defaults to `None`) - Whether to truncate the input texts to fit within the context length.
- If `True`, over-length input texts will be truncated to fit within the context length, before vectorized by the embedding model.
- If `False`, an error will be raised if any given text exceeds the context length.
- If not specified (defaults to `None`), Voyage will truncate the input text before sending it to the embedding model if it slightly exceeds the context window length. If it significantly exceeds the context window length, an error will be raised.
### Voyage HTTP API
You can also get embeddings by requesting Voyage HTTP API. For example, you can send an HTTP request through the `curl` command in a terminal:
```bash
curl https://api.voyageai.com/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $VOYAGE_API_KEY" \
-d '{
"input": ["Sample text 1", "Sample text 2"],
"model": "voyage-2"
}'
```
The response you would get is a JSON object containing the embeddings and the token usage:
```bash
{
"object": "list",
"data": [
{
"embedding": [0.02012746, 0.01957859, ...],
"index": 0
},
{
"embedding": [0.01429677, 0.03077182, ...],
"index": 1
}
],
"model": "voyage-2",
"usage": {
"total_tokens": 10
}
}
```
Voyage AI's embedding endpoint is `https://api.voyageai.com/v1/embeddings` (POST). The request header must contain the API key. The request body is a JSON object containing the following arguments:
- **input** (str, List[str]) - A single text string, or a list of texts as a list of strings. Currently, the maximum length of the list is 128, and total number of tokens in the list is at most 320K for `voyage-2` and 120K for `voyage-code-2`.
- **model** (str) - Name of the model. Recommended options: `voyage-2` (default), `voyage-code-2`.
- **input_type** (str, optional, defaults to `None`) - Type of the input text. Defalut to `None`. Other options: `query`, `document`.
- **truncation** (bool, optional, defaults to `None`) - Whether to truncate the input texts to fit within the context length.
- If `True`, over-length input texts will be truncated to fit within the context length, before vectorized by the embedding model.
- If `False`, an error will be raised if any given text exceeds the context length.
- If not specified (defaults to `None`), Voyage will truncate the input text before sending it to the embedding model if it slightly exceeds the context window length. If it significantly exceeds the context window length, an error will be raised.
- **encoding_format** (str, optional, default to `None`) - Format in which the embeddings are encoded. Voyage currently supports two options:
- If not specified (defaults to `None`): the embeddings are represented as lists of floating-point numbers;
- `"base64"`: the embeddings are compressed to [Base64](https://docs.python.org/3/library/base64.html) encodings.
### AWS Marketplace
Voyage embeddings are available on [AWS Marketplace](https://aws.amazon.com/marketplace/seller-profile?id=seller-snt4gb6fd7ljg). Here is the instruction for accessing Voyage on AWS:
1. Subscribe to the model package
1. Navigate to the [model package listing page](https://aws.amazon.com/marketplace/seller-profile?id=seller-snt4gb6fd7ljg) and select the model to deploy.
1. Click on the *Continue to subscribe* button.
1. On the *Subscribe to this software* page, please carefully review the details. If you and your organization agree with the standard End-User License Agreement (EULA), pricing, and support terms, click on "Accept Offer".
1. After selecting *Continue to configuration* and choosing a region, you will be presented with a Product Arn. This is the model package ARN required for creating a deployable model using Boto3. Copy the ARN that corresponds to your selected region and use it in the subsequent cell.
2. Deploy the model package
From now on, we recommend you to continue with our provided [notebook](https://github.com/voyage-ai/voyageai-aws/blob/main/notebooks/deploy_voyage_code_2_sagemaker.ipynb) in [Sagemaker Studio](https://aws.amazon.com/sagemaker/studio/). Please create a JupyterLab space, upload our notebook, and continue from there.
## Available Models
Voyage recommends using the following embedding models:
| Model | Context Length | Embedding Dimension | Description |
| --- | --- | --- | --- |
| `voyage-2` | 4000 | 1024 | Latest base (generalist) embedding model with the best retrieval quality. See [blog post](https://blog.voyageai.com/2023/10/29/voyage-embeddings/?ref=anthropic) for details. |
| `voyage-code-2` | 16000 | 1536 | Optimized for code retrieval (17% better than alternatives), and also SoTA on general-purpose corpora. See [blog post](https://blog.voyageai.com/2024/01/23/voyage-code-2-elevate-your-code-retrieval/?ref=anthropic) for details. |
`voyage-2` is a generalist embedding model, which achieves state-of-the-art performance across domains and retains high efficiency. `voyage-code-2` is optimized for code applications, offering 4x the context length for more flexible usage, albeit at a slightly higher latency.
Voyage is actively developing more advanced and specialized models, and can fine-tune embeddings for your company. Please email [contact@voyageai.com](mailto:contact@voyageai.com) for trial access or finetuning on your own data!
- `voyage-finance-2`: coming soon
- `voyage-law-2`: coming soon
- `voyage-multilingual-2`: coming soon
- `voyage-healthcare-2`: coming soon
## Motivating Example
Now that we know how to get embeddings, let's see a brief motivating example.
Suppose we have a small corpus of six documents to retrieve from
```python
documents = [
"The Mediterranean diet emphasizes fish, olive oil, and vegetables, believed to reduce chronic diseases.",
"Photosynthesis in plants converts light energy into glucose and produces essential oxygen.",
"20th-century innovations, from radios to smartphones, centered on electronic advancements.",
"Rivers provide water, irrigation, and habitat for aquatic species, vital for ecosystems.",
"Apple’s conference call to discuss fourth fiscal quarter results and business updates is scheduled for Thursday, November 2, 2023 at 2:00 p.m. PT / 5:00 p.m. ET.",
"Shakespeare's works, like 'Hamlet' and 'A Midsummer Night's Dream,' endure in literature."
]
```
We will first use Voyage to convert each of them into an embedding vector
```python
import voyageai
vo = voyageai.Client()
# Embed the documents
doc_embds = vo.embed(
documents, model="voyage-2", input_type="document"
).embeddings
```
The embeddings will allow us to do semantic search / retrieval in the vector space. Given an example query,
```python
query = "When is Apple's conference call scheduled?"
```
we convert it into an embedding, and conduct a nearest neighbor search to find the most relevant document based on the distance in the embedding space.
```python
import numpy as np
# Embed the query
query_embd = vo.embed(
[query], model="voyage-2", input_type="query"
).embeddings[0]
# Compute the similarity
# Voyage embeddings are normalized to length 1, therefore dot-product
# and cosine similarity are the same.
similarities = np.dot(doc_embds, query_embd)
retrieved_id = np.argmax(similarities)
print(documents[retrieved_id])
```
Note that we use `input_type="document"` and `input_type="query"` for embedding the document and query, respectively. More specification can be found [here](#voyage-python-package).
The output would be the 5th document, which is indeed the most relevant to the query:
```
Apple’s conference call to discuss fourth fiscal quarter results and business updates is scheduled for Thursday, November 2, 2023 at 2:00 p.m. PT / 5:00 p.m. ET.
```
If you are looking for a detailed set of cookbooks on how to do RAG with embeddings, including vector databases, check out our [RAG cookbook](https://github.com/anthropics/anthropic-cookbook/blob/main/third_party/Pinecone/rag_using_pinecone.ipynb).
## Frequently Asked Questions
### How do I calculate the distance between two embedding vectors?
Cosine similarity is a popular choice, but most distance functions will do fine. Voyage embeddings are normalized to length 1, therefore cosine similarity is essentially the same as the dot-product between two vectors. Here is a code snippet you can use for calculating cosine similarity between two embedding vectors.
```python
import numpy
similarity = np.dot(embd1, embd2)
# Voyage embeddings are normalized to length 1, therefore cosine similarity
# is the same as dot-product.
```
If you want to find the K nearest embedding vectors over a large corpus, we recommend using the capabilities built into most vector databases.
### Can I count the number of tokens in a string before embedding it?
Yes! You can do so with the following code.
```python
import voyageai
vo = voyageai.Client()
total_tokens = vo.count_tokens(["Sample text"])
```
## Pricing
Pricing information is available on the Voyage website's [pricing page](https://docs.voyageai.com/pricing/?ref=anthropic), and should be checked there.
---
## Iteratively Searching Wikipedia with Claude
Source: https://github.com/anthropics/claude-cookbooks/blob/main/third_party/Wikipedia/wikipedia-search-cookbook.ipynb
Path: 11-RAG-Search/claude-cookbooks-third-party-wikipedia-wikipedia-search-cookbook.md
# Iteratively Searching Wikipedia with Claude
:::
::: {.cell .markdown}
\[DISCLAIMER: This notebook was created using Claude 2 models and is
considered legacy.\]
Some questions can\'t be answered by Claude off the top of Claude\'s
head. Maybe they\'re about current events. Maybe you have an intensely
detailed question that Claude hasn\'t memorized the answer to. No
worries! With some prompting and scaffolding, Claude can search the web
to find answers. In this notebook, we will create a virtual research
assistant who has the ability to search Wikipedia to find answers to
your question. The same approach can be used to allow Claude to search
the broader web, or a set of documents you provide.
What is the approach? Broadly it falls under the category of \"tool
use\". We create a search tool, tell Claude about it, and let it go to
work. In pseudocode:
1. Prompt Claude with a description of the search tool, how it\'s best
used, and how to \"call\" it (by issuing a special string).
2. Tell Claude your question.
3. Claude produces tokens like normal. If it produces the special
string, terminate the token production stream, and issue a query to
a search API.
4. Construct a new prompt which consists of the prompt from step 1,
plus everything Claude generated up to the search call string, plus
the results of the API call.
5. Repeat until Claude decides it\'s done.
Let\'s zoom in on the prompts for tool use and retrieval.
:::
::: {.cell .markdown}
### Prompts
:::
:::: {.cell .code execution_count="101"}
``` python
# Tool Description Prompt
wikipedia_prompt = """You will be asked a question by a human user. You have access to the following tool to help answer the question. <tool_description> Search Engine Tool * The search engine will exclusively search over Wikipedia for pages similar to your query. It returns for each page its title and full page content. Use this tool if you want to get up-to-date and comprehensive information on a topic to help answer queries. Queries should be as atomic as possible -- they only need to address one part of the user's question. For example, if the user's query is "what is the color of a basketball?", your search query should be "basketball". Here's another example: if the user's question is "Who created the first neural network?", your first query should be "neural network". As you can see, these queries are quite short. Think keywords, not phrases. * At any time, you can make a call to the search engine using the following syntax: <search_query>query_word</search_query>. * You'll then get results back in <search_result> tags.</tool_description>"""
print(wikipedia_prompt)
```
::: {.output .stream .stdout}
You will be asked a question by a human user. You have access to the following tool to help answer the question. <tool_description> Search Engine Tool * The search engine will exclusively search over Wikipedia for pages similar to your query. It returns for each page its title and full page content. Use this tool if you want to get up-to-date and comprehensive information on a topic to help answer queries. Queries should be as atomic as possible -- they only need to address one part of the user's question. For example, if the user's query is "what is the color of a basketball?", your search query should be "basketball". Here's another example: if the user's question is "Who created the first neural network?", your first query should be "neural network". As you can see, these queries are quite short. Think keywords, not phrases. * At any time, you can make a call to the search engine using the following syntax: <search_query>query_word</search_query>. * You'll then get results back in <search_result> tags.</tool_description>
:::
::::
::: {.cell .markdown}
Notice that there is a lot of advice in this prompt about how to search
Wikipedia properly. We\'re all used to just typing random nonsense into
Google and getting decent results because the query parsing logic is so
good. Wikipedia search is not like that. As an example: consider the
query \"What\'s the best way to purchase potatoes in the United Arab
Emirates\". The [top hits for this on
Wikipedia](https://en.wikipedia.org/w/index.php?search=What%27s+the+best+way+to+purchase+potatoes+in+the+United+Arab+Emirates&title=Special:Search&profile=advanced&fulltext=1&ns0=1)
are for Slavery in the United States, 1973 Oil Crisis, Wendy\'s, and Tim
Horton\'s (??). Meanwhile Google correctly takes you straight to
Carrefour UAE.
:::
::: {.cell .markdown}
Another difference is that Wikipedia search returns entire pages. With
vector search, you might be getting narrower chunks, so you might want
to ask for more results, use a more specific query, or both. The
big-picture takeaway is that your results can vary a lot on your choices
here so pay attention!
:::
:::: {.cell .code execution_count="103"}
``` python
retrieval_prompt = """Before beginning to research the user's question, first think for a moment inside <scratchpad> tags about what information is necessary for a well-informed answer. If the user's question is complex, you may need to decompose the query into multiple subqueries and execute them individually. Sometimes the search engine will return empty search results, or the search results may not contain the information you need. In such cases, feel free to try again with a different query.
After each call to the Search Engine Tool, reflect briefly inside <search_quality></search_quality> tags about whether you now have enough information to answer, or whether more information is needed. If you have all the relevant information, write it in <information></information> tags, WITHOUT actually answering the question. Otherwise, issue a new search.
Here is the user's question: <question>{query}</question> Remind yourself to make short queries in your scratchpad as you plan out your strategy."""
print(retrieval_prompt)
```
::: {.output .stream .stdout}
Before beginning to research the user's question, first think for a moment inside <scratchpad> tags about what information is necessary for a well-informed answer. If the user's question is complex, you may need to decompose the query into multiple subqueries and execute them individually. Sometimes the search engine will return empty search results, or the search results may not contain the information you need. In such cases, feel free to try again with a different query.
After each call to the Search Engine Tool, reflect briefly inside <search_quality></search_quality> tags about whether you now have enough information to answer, or whether more information is needed. If you have all the relevant information, write it in <information></information> tags, WITHOUT actually answering the question. Otherwise, issue a new search.
Here is the user's question: <question>{query}</question> Remind yourself to make short queries in your scratchpad as you plan out your strategy.
:::
::::
::: {.cell .markdown}
We use a scratchpad here for the normal chain-of-thought reasons \-- it
makes Claude come up with a coherent plan to answer the question. The
search quality reflection is used to induce Claude to be persistent and
not jump the gun by answering the question before gathering all the
relevant information. But why are we telling Claude to synthesize the
information and not answer right away?
:::
:::: {.cell .code execution_count="104"}
``` python
answer_prompt = "Here is a user query: <query>{query}</query>. Here is some relevant information: <information>{information}</information>. Please answer the question using the relevant information."
print(answer_prompt)
```
::: {.output .stream .stdout}
Here is a user query: <query>{query}</query>. Here is some relevant information: <information>{information}</information>. Please answer the question using the relevant information.
:::
::::
::: {.cell .markdown}
By extracting the information and presenting it to Claude in a new
query, we allow Claude to focus all its attention on synthesizing the
information into the right answer. Without this step, we found that
Claude would sometimes precommit to an answer and then \"justify\" it
with the search results, rather than allowing the results to guide it.
:::
::: {.cell .markdown}
Now follows a bunch of code that implements the pseudocode for
searching + retrieving + reprompting.
### Search Implementation
:::
::: {.cell .code execution_count="88"}
``` python
import re
from abc import abstractmethod
from dataclasses import dataclass
import wikipedia
from anthropic import AI_PROMPT, HUMAN_PROMPT, Anthropic
@dataclass
class SearchResult:
"""
A single search result.
"""
content: str
class SearchTool:
"""
A search tool that can run a query and return a formatted string of search results.
"""
def __init__():
pass
@abstractmethod
def raw_search(self, query: str, n_search_results_to_use: int) -> list[SearchResult]:
"""
Runs a query using the searcher, then returns the raw search results without formatting.
:param query: The query to run.
:param n_search_results_to_use: The number of results to return.
"""
raise NotImplementedError()
@abstractmethod
def process_raw_search_results(
self,
results: list[SearchResult],
) -> list[str]:
"""
Extracts the raw search content from the search results and returns a list of strings that can be passed to Claude.
:param results: The search results to extract.
"""
raise NotImplementedError()
def search_results_to_string(self, extracted: list[str]) -> str:
"""
Joins and formats the extracted search results as a string.
:param extracted: The extracted search results to format.
"""
result = "\n".join(
[
f'<item index="{i + 1}">\n<page_content>\n{r}\n</page_content>\n</item>'
for i, r in enumerate(extracted)
]
)
return result
def wrap_search_results(self, extracted: list[str]) -> str:
"""
Formats the extracted search results as a string, including the <search_results> tags.
:param extracted: The extracted search results to format.
"""
return f"\n<search_results>\n{self.search_results_to_string(extracted)}\n</search_results>"
def search(self, query: str, n_search_results_to_use: int) -> str:
raw_search_results = self.raw_search(query, n_search_results_to_use)
processed_search_results = self.process_raw_search_results(raw_search_results)
displayable_search_results = self.wrap_search_results(processed_search_results)
return displayable_search_results
```
:::
::: {.cell .code}
``` python
@dataclass
class WikipediaSearchResult(SearchResult):
title: str
class WikipediaSearchTool(SearchTool):
def __init__(self, truncate_to_n_tokens: int | None = 5000):
self.truncate_to_n_tokens = truncate_to_n_tokens
if truncate_to_n_tokens is not None:
self.tokenizer = Anthropic().get_tokenizer()
def raw_search(self, query: str, n_search_results_to_use: int) -> list[WikipediaSearchResult]:
search_results = self._search(query, n_search_results_to_use)
return search_results
def process_raw_search_results(self, results: list[WikipediaSearchResult]) -> list[str]:
processed_search_results = [
f"Page Title: {result.title.strip()}\nPage Content:\n{self.truncate_page_content(result.content)}"
for result in results
]
return processed_search_results
def truncate_page_content(self, page_content: str) -> str:
if self.truncate_to_n_tokens is None:
return page_content.strip()
else:
return self.tokenizer.decode(
self.tokenizer.encode(page_content).ids[: self.truncate_to_n_tokens]
).strip()
def _search(self, query: str, n_search_results_to_use: int) -> list[WikipediaSearchResult]:
results: list[str] = wikipedia.search(query)
search_results: list[WikipediaSearchResult] = []
for result in results:
if len(search_results) >= n_search_results_to_use:
break
try:
page = wikipedia.page(result)
print(page.url)
except wikipedia.exceptions.WikipediaException:
# The Wikipedia API is a little flaky, so we just skip over pages that fail to load
continue
content = page.content
title = page.title
search_results.append(WikipediaSearchResult(content=content, title=title))
return search_results
```
:::
::: {.cell .code execution_count="100"}
``` python
def extract_between_tags(tag: str, string: str, strip: bool = True) -> list[str]:
ext_list = re.findall(rf"<{tag}\s?>(.+?)</{tag}\s?>", string, re.DOTALL)
if strip:
ext_list = [e.strip() for e in ext_list]
return ext_list
class ClientWithRetrieval(Anthropic):
def __init__(self, search_tool: SearchTool, verbose: bool = True, *args, **kwargs):
super().__init__(*args, **kwargs)
self.search_tool = search_tool
self.verbose = verbose
# Helper methods
def _search_query_stop(
self, partial_completion: str, n_search_results_to_use: int
) -> tuple[list[SearchResult], str]:
search_query = extract_between_tags("search_query", partial_completion + "</search_query>")
if search_query is None:
raise Exception(
"Completion with retrieval failed as partial completion returned mismatched <search_query> tags."
)
print(f"Running search query against SearchTool: {search_query}")
search_results = self.search_tool.raw_search(search_query, n_search_results_to_use)
extracted_search_results = self.search_tool.process_raw_search_results(search_results)
formatted_search_results = self.search_tool.wrap_search_results(extracted_search_results)
return search_results, formatted_search_results
def retrieve(
self,
query: str,
model: str,
n_search_results_to_use: int = 3,
stop_sequences: list[str] = None,
max_tokens_to_sample: int = 1000,
max_searches_to_try: int = 5,
temperature: float = 1.0,
) -> tuple[list[SearchResult], str]:
if stop_sequences is None:
stop_sequences = [HUMAN_PROMPT]
prompt = (
f"{HUMAN_PROMPT} {wikipedia_prompt} {retrieval_prompt.format(query=query)}{AI_PROMPT}"
)
starting_prompt = prompt
print("Starting prompt:", starting_prompt)
token_budget = max_tokens_to_sample
all_raw_search_results: list[SearchResult] = []
for tries in range(max_searches_to_try):
partial_completion = self.completions.create(
prompt=prompt,
stop_sequences=stop_sequences + ["</search_query>"],
model=model,
max_tokens_to_sample=token_budget,
temperature=temperature,
)
partial_completion, stop_reason, stop_seq = (
partial_completion.completion,
partial_completion.stop_reason,
partial_completion.stop,
)
print(partial_completion)
token_budget -= self.count_tokens(partial_completion)
prompt += partial_completion
if stop_reason == "stop_sequence" and stop_seq == "</search_query>":
print(f"Attempting search number {tries}.")
raw_search_results, formatted_search_results = self._search_query_stop(
partial_completion, n_search_results_to_use
)
prompt += "</search_query>" + formatted_search_results
all_raw_search_results += raw_search_results
else:
break
final_model_response = prompt[len(starting_prompt) :]
return all_raw_search_results, final_model_response
# Main methods
def completion_with_retrieval(
self,
query: str,
model: str,
n_search_results_to_use: int = 3,
stop_sequences: list[str] = None,
max_tokens_to_sample: int = 1000,
max_searches_to_try: int = 5,
temperature: float = 1.0,
) -> str:
if stop_sequences is None:
stop_sequences = [HUMAN_PROMPT]
_, retrieval_response = self.retrieve(
query,
model=model,
n_search_results_to_use=n_search_results_to_use,
stop_sequences=stop_sequences,
max_tokens_to_sample=max_tokens_to_sample,
max_searches_to_try=max_searches_to_try,
temperature=temperature,
)
information = extract_between_tags("information", retrieval_response)[-1]
prompt = f"{HUMAN_PROMPT} {answer_prompt.format(query=query, information=information)}{AI_PROMPT}"
print("Summarizing:\n", prompt)
answer = self.completions.create(
prompt=prompt, model=model, temperature=temperature, max_tokens_to_sample=1000
).completion
return answer
```
:::
::: {.cell .markdown}
### Running a Query
:::
::: {.cell .markdown}
We\'re ready to execute a query! Let\'s pick something:
- recent, so it\'s less likely to be in Claude\'s training data, and
- compound/complex so it requires multiple searches.
:::
:::: {.cell .code execution_count="98"}
``` python
import os
# Create a searcher
wikipedia_search_tool = WikipediaSearchTool()
ANTHROPIC_SEARCH_MODEL = "claude-2"
client = ClientWithRetrieval(
api_key=os.environ["ANTHROPIC_API_KEY"], verbose=True, search_tool=wikipedia_search_tool
)
query = "Which movie came out first: Oppenheimer, or Are You There God It's Me Margaret?"
augmented_response = client.completion_with_retrieval(
query=query,
model=ANTHROPIC_SEARCH_MODEL,
n_search_results_to_use=1,
max_searches_to_try=5,
max_tokens_to_sample=1000,
temperature=0,
)
print(augmented_response)
```
::: {.output .stream .stdout}
Starting prompt:
Human: You will be asked a question by a human user. You have access to the following tool to help answer the question. <tool_description> Search Engine Tool * The search engine will exclusively search over Wikipedia for pages similar to your query. It returns for each page its title and full page content. Use this tool if you want to get up-to-date and comprehensive information on a topic to help answer queries. Queries should be as atomic as possible -- they only need to address one part of the user's question. For example, if the user's query is "what is the color of a basketball?", your search query should be "basketball". Here's another example: if the user's question is "Who created the first neural network?", your first query should be "neural network". As you can see, these queries are quite short. Think keywords, not phrases. * At any time, you can make a call to the search engine using the following syntax: <search_query>query_word</search_query>. * You'll then get results back in <search_result> tags.</tool_description> Before beginning to research the user's question, first think for a moment inside <scratchpad> tags about what information is necessary for a well-informed answer. If the user's question is complex, you may need to decompose the query into multiple subqueries and execute them individually. Sometimes the search engine will return empty search results, or the search results may not contain the information you need. In such cases, feel free to try again with a different query.
After each call to the Search Engine Tool, reflect briefly inside <search_quality></search_quality> tags about whether you now have enough information to answer, or whether more information is needed. If you have all the relevant information, write it in <information></information> tags, WITHOUT actually answering the question. Otherwise, issue a new search.
Here is the user's question: <question>Which movie came out first: Oppenheimer, or Are You There God It's Me Margaret?</question> Remind yourself to make short queries in your scratchpad as you plan out your strategy.
Assistant:
<scratchpad>
To answer this question, I need to find the release dates for the two movies:
- Oppenheimer release date
- Are You There God It's Me Margaret release date
I can search for each movie title individually to get the release date.
</scratchpad>
<search_query>Oppenheimer movie
Attempting search number 0.
Running search query against SearchTool: ['Oppenheimer movie']
https://en.wikipedia.org/wiki/Oppenheimer_(film)
The search results indicate that Oppenheimer is scheduled for theatrical release on July 21, 2023. This provides the release date for Oppenheimer.
<search_quality>The search results directly provided the release date for Oppenheimer, so I now have enough information to answer this part of the question.</search_quality>
<search_query>Are You There God It's Me Margaret movie
Attempting search number 1.
Running search query against SearchTool: ["Are You There God It's Me Margaret movie"]
https://en.wikipedia.org/wiki/Are_You_There_God%3F_It%27s_Me,_Margaret.
The search results indicate that the film adaptation of Are You There God? It's Me, Margaret was released on April 28, 2023. This provides the release date for Are You There God? It's Me, Margaret.
<search_quality>The search results directly stated the release date for the Are You There God? It's Me, Margaret movie adaptation, so I now have enough information to fully answer the question.</search_quality>
<information>
- Oppenheimer was released on July 21, 2023
- Are You There God? It's Me, Margaret was released on April 28, 2023
</information>
Based on the release dates found through my searches, Oppenheimer came out first, being released on July 21, 2023, while Are You There God? It's Me, Margaret was released later on April 28, 2023.
Summarizing:
Human:Here is a user query: <query>Which movie came out first: Oppenheimer, or Are You There God It's Me Margaret?</query>. Here is some relevant information: <information>- Oppenheimer was released on July 21, 2023
- Are You There God? It's Me, Margaret was released on April 28, 2023</information>. Please answer the question using the relevant information.
Assistant:
Based on the information provided, Are You There God? It's Me, Margaret was released first, on April 28, 2023. Oppenheimer was released later, on July 21, 2023.
:::
::::
::: {.cell .markdown}
Cool, Claude was able to make a plan, execute the queries, and
synthesize the information into an accurate answer. Note: without the
extra information extraction step, Claude would sometimes determine the
release dates of the movies correctly but then get the ordering wrong in
its final answer. Let\'s do another.
:::
:::: {.cell .code execution_count="99"}
``` python
augmented_response = client.completion_with_retrieval(
query="Who won the 2023 NBA championship? Who was that team's best player in the year 2009?",
model=ANTHROPIC_SEARCH_MODEL,
n_search_results_to_use=1,
max_searches_to_try=5,
max_tokens_to_sample=1000,
temperature=0,
)
print(augmented_response)
```
::: {.output .stream .stdout}
Starting prompt:
Human: You will be asked a question by a human user. You have access to the following tool to help answer the question. <tool_description> Search Engine Tool * The search engine will exclusively search over Wikipedia for pages similar to your query. It returns for each page its title and full page content. Use this tool if you want to get up-to-date and comprehensive information on a topic to help answer queries. Queries should be as atomic as possible -- they only need to address one part of the user's question. For example, if the user's query is "what is the color of a basketball?", your search query should be "basketball". Here's another example: if the user's question is "Who created the first neural network?", your first query should be "neural network". As you can see, these queries are quite short. Think keywords, not phrases. * At any time, you can make a call to the search engine using the following syntax: <search_query>query_word</search_query>. * You'll then get results back in <search_result> tags.</tool_description> Before beginning to research the user's question, first think for a moment inside <scratchpad> tags about what information is necessary for a well-informed answer. If the user's question is complex, you may need to decompose the query into multiple subqueries and execute them individually. Sometimes the search engine will return empty search results, or the search results may not contain the information you need. In such cases, feel free to try again with a different query.
After each call to the Search Engine Tool, reflect briefly inside <search_quality></search_quality> tags about whether you now have enough information to answer, or whether more information is needed. If you have all the relevant information, write it in <information></information> tags, WITHOUT actually answering the question. Otherwise, issue a new search.
Here is the user's question: <question>Who won the 2023 NBA championship? Who was that team's best player in the year 2009?</question> Remind yourself to make short queries in your scratchpad as you plan out your strategy.
Assistant:
<scratchpad>
To answer this question, I need to find:
1. The team that won the 2023 NBA championship
2. The best player on that team in 2009
I can search for these things separately.
</scratchpad>
<search_query>2023 nba championship winner
Attempting search number 0.
Running search query against SearchTool: ['2023 nba championship winner']
https://en.wikipedia.org/wiki/List_of_NBA_champions
<search_quality>The search results contain the team that won the 2023 NBA championship, so I have the information I need to answer the first part of the question.</search_quality>
<information>
The Denver Nuggets won the 2023 NBA championship.
</information>
<search_query>denver nuggets best player 2009
Attempting search number 1.
Running search query against SearchTool: ['denver nuggets best player 2009']
https://en.wikipedia.org/wiki/2009%E2%80%9310_Denver_Nuggets_season
<search_quality>The search results indicate that Carmelo Anthony was the Nuggets' best player in 2009, so I now have all the information needed to fully answer the question.</search_quality>
<information>
- The Denver Nuggets won the 2023 NBA championship.
- Carmelo Anthony was the Nuggets' best player in 2009.
</information>
Summarizing:
Human:Here is a user query: <query>Who won the 2023 NBA championship? Who was that team's best player in the year 2009?</query>. Here is some relevant information: <information>- The Denver Nuggets won the 2023 NBA championship.
- Carmelo Anthony was the Nuggets' best player in 2009.</information>. Please answer the question using the relevant information.
Assistant:
<response>
Based on the provided information:
- The Denver Nuggets won the 2023 NBA championship.
- Carmelo Anthony was the Nuggets' best player in 2009.
</response>
:::
::::
::: {.cell .markdown}
And there you have it! You may notice that the search tool code is nice
and abstract and can be adapted to use a search API of your choice with
minor modifications. Just remember to explain to Claude any tips it
needs to use the tool well. You can even give Claude some few-shot
examples of ideal query plans and query structure to improve performance
further.
:::
---
## Extracting Structured JSON using Claude and Tool Use
Source: https://github.com/anthropics/claude-cookbooks/blob/main/tool_use/extracting_structured_json.ipynb
Path: 11-RAG-Search/claude-cookbooks-tool-use-extracting-structured-json.md
# Extracting Structured JSON using Claude and Tool Use
In this cookbook, we\'ll explore various examples of using Claude and
the tool use feature to extract structured JSON data from different
types of input. We\'ll define custom tools that prompt Claude to
generate well-structured JSON output for tasks such as summarization,
entity extraction, sentiment analysis, and more.
If you want to get structured JSON data without using tools, take a look
at our \"[How to enable JSON
mode](https://github.com/anthropics/anthropic-cookbook/blob/main/misc/how_to_enable_json_mode.ipynb)\"
cookbook.
:::
::: {.cell .markdown}
## Set up the environment
First, let\'s install the required libraries and set up the Claude API
client.
:::
::: {.cell .code}
``` python
%pip install anthropic requests beautifulsoup4
```
:::
::: {.cell .code execution_count="54"}
``` python
import json
import requests
from anthropic import Anthropic
from bs4 import BeautifulSoup
client = Anthropic()
MODEL_NAME = "claude-haiku-4-5"
```
:::
::: {.cell .markdown}
## Example 1: Article Summarization
In this example, we\'ll use Claude to generate a JSON summary of an
article, including fields for the author, topics, summary, coherence
score, persuasion score, and a counterpoint.
:::
::: {.cell .code}
``` python
tools = [
{
"name": "print_summary",
"description": "Prints a summary of the article.",
"input_schema": {
"type": "object",
"properties": {
"author": {"type": "string", "description": "Name of the article author"},
"topics": {
"type": "array",
"items": {"type": "string"},
"description": 'Array of topics, e.g. ["tech", "politics"]. Should be as specific as possible, and can overlap.',
},
"summary": {
"type": "string",
"description": "Summary of the article. One or two paragraphs max.",
},
"coherence": {
"type": "integer",
"description": "Coherence of the article's key points, 0-100 (inclusive)",
},
"persuasion": {
"type": "number",
"description": "Article's persuasion score, 0.0-1.0 (inclusive)",
},
},
"required": ["author", "topics", "summary", "coherence", "persuasion", "counterpoint"],
},
}
]
url = "https://www.anthropic.com/news/third-party-testing"
response = requests.get(url, timeout=30)
soup = BeautifulSoup(response.text, "html.parser")
article = " ".join([p.text for p in soup.find_all("p")])
query = f"""
<article>
{article}
</article>
Use the `print_summary` tool.
"""
response = client.messages.create(
model=MODEL_NAME, max_tokens=4096, tools=tools, messages=[{"role": "user", "content": query}]
)
json_summary = None
for content in response.content:
if content.type == "tool_use" and content.name == "print_summary":
json_summary = content.input
break
if json_summary:
print("JSON Summary:")
print(json.dumps(json_summary, indent=2))
else:
print("No JSON summary found in the response.")
```
:::
::: {.cell .markdown}
## Example 2: Named Entity Recognition
In this example, we\'ll use Claude to perform named entity recognition
on a given text and return the entities in a structured JSON format.
:::
:::: {.cell .code execution_count="63"}
``` python
tools = [
{
"name": "print_entities",
"description": "Prints extract named entities.",
"input_schema": {
"type": "object",
"properties": {
"entities": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string", "description": "The extracted entity name."},
"type": {
"type": "string",
"description": "The entity type (e.g., PERSON, ORGANIZATION, LOCATION).",
},
"context": {
"type": "string",
"description": "The context in which the entity appears in the text.",
},
},
"required": ["name", "type", "context"],
},
}
},
"required": ["entities"],
},
}
]
text = "John works at Google in New York. He met with Sarah, the CEO of Acme Inc., last week in San Francisco."
query = f"""
<document>
{text}
</document>
Use the print_entities tool.
"""
response = client.messages.create(
model=MODEL_NAME, max_tokens=4096, tools=tools, messages=[{"role": "user", "content": query}]
)
json_entities = None
for content in response.content:
if content.type == "tool_use" and content.name == "print_entities":
json_entities = content.input
break
if json_entities:
print("Extracted Entities (JSON):")
print(json_entities)
else:
print("No entities found in the response.")
```
::: {.output .stream .stdout}
Extracted Entities (JSON):
{'entities': [{'name': 'John', 'type': 'PERSON', 'context': 'John works at Google in New York.'}, {'name': 'Google', 'type': 'ORGANIZATION', 'context': 'John works at Google in New York.'}, {'name': 'New York', 'type': 'LOCATION', 'context': 'John works at Google in New York.'}, {'name': 'Sarah', 'type': 'PERSON', 'context': 'He met with Sarah, the CEO of Acme Inc., last week in San Francisco.'}, {'name': 'Acme Inc.', 'type': 'ORGANIZATION', 'context': 'He met with Sarah, the CEO of Acme Inc., last week in San Francisco.'}, {'name': 'San Francisco', 'type': 'LOCATION', 'context': 'He met with Sarah, the CEO of Acme Inc., last week in San Francisco.'}]}
:::
::::
::: {.cell .markdown}
## Example 3: Sentiment Analysis
In this example, we\'ll use Claude to perform sentiment analysis on a
given text and return the sentiment scores in a structured JSON format.
:::
:::: {.cell .code execution_count="64"}
``` python
tools = [
{
"name": "print_sentiment_scores",
"description": "Prints the sentiment scores of a given text.",
"input_schema": {
"type": "object",
"properties": {
"positive_score": {
"type": "number",
"description": "The positive sentiment score, ranging from 0.0 to 1.0.",
},
"negative_score": {
"type": "number",
"description": "The negative sentiment score, ranging from 0.0 to 1.0.",
},
"neutral_score": {
"type": "number",
"description": "The neutral sentiment score, ranging from 0.0 to 1.0.",
},
},
"required": ["positive_score", "negative_score", "neutral_score"],
},
}
]
text = "The product was okay, but the customer service was terrible. I probably won't buy from them again."
query = f"""
<text>
{text}
</text>
Use the print_sentiment_scores tool.
"""
response = client.messages.create(
model=MODEL_NAME, max_tokens=4096, tools=tools, messages=[{"role": "user", "content": query}]
)
json_sentiment = None
for content in response.content:
if content.type == "tool_use" and content.name == "print_sentiment_scores":
json_sentiment = content.input
break
if json_sentiment:
print("Sentiment Analysis (JSON):")
print(json.dumps(json_sentiment, indent=2))
else:
print("No sentiment analysis found in the response.")
```
::: {.output .stream .stdout}
Sentiment Analysis (JSON):
{
"negative_score": 0.6,
"neutral_score": 0.3,
"positive_score": 0.1
}
:::
::::
::: {.cell .markdown}
## Example 4: Text Classification
In this example, we\'ll use Claude to classify a given text into
predefined categories and return the classification results in a
structured JSON format.
:::
:::: {.cell .code execution_count="66"}
``` python
tools = [
{
"name": "print_classification",
"description": "Prints the classification results.",
"input_schema": {
"type": "object",
"properties": {
"categories": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string", "description": "The category name."},
"score": {
"type": "number",
"description": "The classification score for the category, ranging from 0.0 to 1.0.",
},
},
"required": ["name", "score"],
},
}
},
"required": ["categories"],
},
}
]
text = "The new quantum computing breakthrough could revolutionize the tech industry."
query = f"""
<document>
{text}
</document>
Use the print_classification tool. The categories can be Politics, Sports, Technology, Entertainment, Business.
"""
response = client.messages.create(
model=MODEL_NAME, max_tokens=4096, tools=tools, messages=[{"role": "user", "content": query}]
)
json_classification = None
for content in response.content:
if content.type == "tool_use" and content.name == "print_classification":
json_classification = content.input
break
if json_classification:
print("Text Classification (JSON):")
print(json.dumps(json_classification, indent=2))
else:
print("No text classification found in the response.")
```
::: {.output .stream .stdout}
Text Classification (JSON):
{
"categories": [
{
"name": "Politics",
"score": 0.1
},
{
"name": "Sports",
"score": 0.1
},
{
"name": "Technology",
"score": 0.7
},
{
"name": "Entertainment",
"score": 0.1
},
{
"name": "Business",
"score": 0.5
}
]
}
:::
::::
::: {.cell .markdown}
## Example 5: Working with unknown keys
In some cases you may not know the exact JSON object shape up front. In
this example we provide an open ended `input_schema` and instruct Claude
via prompting how to interact with the tool.
:::
:::: {.cell .code execution_count="5"}
``` python
tools = [
{
"name": "print_all_characteristics",
"description": "Prints all characteristics which are provided.",
"input_schema": {"type": "object", "additionalProperties": True},
}
]
query = """Given a description of a character, your task is to extract all the characteristics of the character and print them using the print_all_characteristics tool.
The print_all_characteristics tool takes an arbitrary number of inputs where the key is the characteristic name and the value is the characteristic value (age: 28 or eye_color: green).
<description>
The man is tall, with a beard and a scar on his left cheek. He has a deep voice and wears a black leather jacket.
</description>
Now use the print_all_characteristics tool."""
response = client.messages.create(
model=MODEL_NAME,
max_tokens=4096,
tools=tools,
tool_choice={"type": "tool", "name": "print_all_characteristics"},
messages=[{"role": "user", "content": query}],
)
tool_output = None
for content in response.content:
if content.type == "tool_use" and content.name == "print_all_characteristics":
tool_output = content.input
break
if tool_output:
print("Characteristics (JSON):")
print(json.dumps(json_classification, indent=2))
else:
print("Something went wrong.")
```
::: {.output .stream .stdout}
Characteristics (JSON):
{
"height": "tall",
"facial_hair": "beard",
"facial_features": "scar on left cheek",
"voice": "deep voice",
"clothing": "black leather jacket"
}
:::
::::
::: {.cell .markdown}
These examples demonstrate how you can use Claude and the tool use
feature to extract structured JSON data for various natural language
processing tasks. By defining custom tools with specific input schemas,
you can guide Claude to generate well-structured JSON output that can be
easily parsed and utilized in your applications.
:::
---
## Tool Search: Alternate Approaches
Source: https://github.com/anthropics/claude-cookbooks/blob/main/tool_use/tool_search_alternate_approaches.ipynb
Path: 11-RAG-Search/claude-cookbooks-tool-use-tool-search-alternate-approaches.md
# Tool Search: Alternate Approaches
**Recommend first reading see the cookbook: Tool Search with
Embeddings.**
The goal of this cookbook is to show of some alternate approaches to
using tool search (or really \"tool discovery\") with Claude. In this
cookbook we\'ll demonstrate two useful techniques:
1. Tools can be discovered without \"Search\". In this cookbook, we\'ll
include all of the tool names in Claude\'s system prompt and provide
Claude with decribe_tool_tool to load the tool fully into Claude\'s
context.
2. Tools do not have to be passed in the request\'s `tools` list if
they have not been loaded into Claude\'s context yet. This can be a
bit more application complexity to manage, but can allow your
application to keep requests small, even while Claude has potential
access to thousands of tools.
Users have a lot of flexibility to design tool search to keep Claude\'s
context (and Messages requests) as focused as possible.
## Prerequisites
Before following this guide, ensure you have:
**Required Knowledge**
- Python fundamentals - comfortable with functions, dictionaries, and
basic data structures
- Basic understanding of Claude tool use - we recommend reading the
[Tool Use
Guide](https://docs.anthropic.com/en/docs/build-with-claude/tool-use)
first
**Required Tools**
- Python 3.11 or higher
- Anthropic API key ([get one
here](https://docs.anthropic.com/claude/reference/getting-started-with-the-api))
:::
::: {.cell .code}
``` python
%pip install -q anthropic python-dotenv
```
:::
::: {.cell .code}
``` python
import anthropic
import json
from dotenv import load_dotenv
load_dotenv()
MODEL = "claude-sonnet-4-5-20250929"
client = anthropic.Anthropic()
print("✓ Client initialized")
```
:::
::: {.cell .markdown}
## Define Tool Library
We\'ll define 5 simple tools. In production, this could be hundreds or
thousands of tools stored in a database or configuration file.
:::
::: {.cell .code}
``` python
TOOL_LIBRARY = {
"get_weather": {
"name": "get_weather",
"description": "Get current weather for a city",
"input_schema": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name"},
},
"required": ["city"],
},
},
"get_stock_price": {
"name": "get_stock_price",
"description": "Get current stock price for a ticker symbol",
"input_schema": {
"type": "object",
"properties": {
"ticker": {"type": "string", "description": "Stock ticker (e.g., AAPL)"},
},
"required": ["ticker"],
},
},
"convert_currency": {
"name": "convert_currency",
"description": "Convert amount between currencies",
"input_schema": {
"type": "object",
"properties": {
"amount": {"type": "number"},
"from_currency": {"type": "string"},
"to_currency": {"type": "string"},
},
"required": ["amount", "from_currency", "to_currency"],
},
},
"calculate_tip": {
"name": "calculate_tip",
"description": "Calculate tip amount for a bill",
"input_schema": {
"type": "object",
"properties": {
"bill_amount": {"type": "number"},
"tip_percent": {"type": "number", "default": 20},
},
"required": ["bill_amount"],
},
},
"send_email": {
"name": "send_email",
"description": "Send an email to a recipient",
"input_schema": {
"type": "object",
"properties": {
"to": {"type": "string", "description": "Recipient email"},
"subject": {"type": "string"},
"body": {"type": "string"},
},
"required": ["to", "subject", "body"],
},
},
}
print(f"✓ Defined {len(TOOL_LIBRARY)} tools: {list(TOOL_LIBRARY.keys())}")
```
:::
::: {.cell .markdown}
## The `describe_tool` Tool
Instead of semantic search, we give Claude a simple `describe_tool`
tool. Claude calls this with a tool name to load that tool into context.
The system prompt lists all available tool names, so Claude knows
what\'s available without needing embeddings or search.
:::
::: {.cell .code}
``` python
DESCRIBE_TOOL = {
"name": "describe_tool",
"description": "Load a tool's full definition into context. Call this before using any tool for the first time.",
"input_schema": {
"type": "object",
"properties": {
"tool_name": {
"type": "string",
"description": "Name of the tool to load",
},
},
"required": ["tool_name"],
},
}
# Build system prompt with tool catalog
tool_names = list(TOOL_LIBRARY.keys())
SYSTEM_PROMPT = f"""You are a helpful assistant with access to various tools.
Available tools: {', '.join(tool_names)}
Before using any tool, you must first call describe_tool with the tool name to load it."""
print("System prompt:")
print(SYSTEM_PROMPT)
```
:::
::: {.cell .markdown}
## Mock Tool Execution
Simple mock responses for demonstration:
:::
::: {.cell .code}
``` python
def execute_tool(name: str, inputs: dict) -> str:
"""Mock tool execution."""
if name == "get_weather":
return json.dumps({"city": inputs["city"], "temp": "72°F", "conditions": "Sunny"})
elif name == "get_stock_price":
return json.dumps({"ticker": inputs["ticker"], "price": 185.50, "change": "+1.2%"})
elif name == "convert_currency":
rate = 0.92 if inputs["to_currency"] == "EUR" else 1.0
converted = inputs["amount"] * rate
return json.dumps({"converted": round(converted, 2), "to": inputs["to_currency"]})
elif name == "calculate_tip":
tip = inputs["bill_amount"] * (inputs.get("tip_percent", 20) / 100)
return json.dumps({"tip": round(tip, 2), "total": round(inputs["bill_amount"] + tip, 2)})
elif name == "send_email":
return json.dumps({"status": "sent", "to": inputs["to"]})
return json.dumps({"error": f"Unknown tool: {name}"})
print("✓ Mock execution ready")
```
:::
::: {.cell .markdown}
## Conversation Loop with Dynamic Tool Loading
The key pattern here:
1. **Start with only `describe_tool`** in the tools list
2. **When Claude calls `describe_tool`**, return a `tool_reference` AND
add the tool to `active_tools` with `defer_loading=True`
3. **`defer_loading=True` is critical** - it keeps the tool definition
out of the cached prompt prefix, avoiding cache invalidation when
tools are discovered
Every time Claude sees a `tool_reference` in the conversation, the full
tool definition is loaded into Claude\'s context at that point in the
conversation.
:::
::: {.cell .code}
``` python
def run_conversation(user_message: str, max_turns: int = 10):
"""Run a conversation with dynamic tool loading."""
print(f"\n{'='*60}")
print(f"USER: {user_message}")
print(f"{'='*60}\n")
messages = [{"role": "user", "content": user_message}]
# Start with ONLY describe_tool - no other tools in the request
active_tools = [DESCRIBE_TOOL]
loaded_tools = set() # Track which tools we've added
for turn in range(max_turns):
print(f"--- Turn {turn + 1} (tools in request: {len(active_tools)}) ---")
response = client.messages.create(
model=MODEL,
max_tokens=1024,
system=SYSTEM_PROMPT,
tools=active_tools,
messages=messages,
extra_headers={"anthropic-beta": "advanced-tool-use-2025-11-20"},
)
messages.append({"role": "assistant", "content": response.content})
if response.stop_reason == "end_turn":
for block in response.content:
if block.type == "text":
print(f"\nASSISTANT: {block.text}")
break
# Process tool calls
tool_results = []
for block in response.content:
if block.type == "text" and block.text:
print(f"ASSISTANT: {block.text}")
elif block.type == "tool_use":
tool_name = block.name
tool_input = block.input
if tool_name == "describe_tool":
requested_tool = tool_input["tool_name"]
print(f"🔍 describe_tool({requested_tool})")
if requested_tool in TOOL_LIBRARY:
# Add tool to active_tools with defer_loading=True
# This is critical for prompt caching!
if requested_tool not in loaded_tools:
tool_def = {**TOOL_LIBRARY[requested_tool], "defer_loading": True}
active_tools.append(tool_def)
loaded_tools.add(requested_tool)
print(f" ✓ Added {requested_tool} to tools (defer_loading=True)")
# Return tool_reference so Claude can use it
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": [{"type": "tool_reference", "tool_name": requested_tool}],
})
else:
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": f"Tool '{requested_tool}' not found.",
})
else:
# Execute discovered tool
print(f"🔧 {tool_name}({json.dumps(tool_input)})")
result = execute_tool(tool_name, tool_input)
print(f" → {result}")
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": result,
})
if tool_results:
messages.append({"role": "user", "content": tool_results})
print(f"\n{'='*60}\n")
print("✓ Conversation loop ready")
```
:::
::: {.cell .markdown}
## Example: Weather Query
Watch how Claude:
1. Sees `get_weather` in the system prompt\'s tool list
2. Calls `describe_tool("get_weather")` to load it
3. Receives the `tool_reference` and can now use the tool
:::
::: {.cell .code}
``` python
run_conversation("What's the weather in Tokyo?")
```
:::
::: {.cell .markdown}
## Example: Multi-Tool Query
Claude can load multiple tools as needed:
:::
::: {.cell .code}
``` python
run_conversation("Convert $100 to EUR, then calculate a 20% tip on a $85 dinner bill.")
```
:::
::: {.cell .markdown}
## Why `defer_loading=True` Matters
When you add a tool to the `tools` list, the tool definition is normally
loaded into the very beginning of Claude\'s context window. If you add
new tools, you will lose most of the cache because the very beginning of
the context window has changed.
**With `defer_loading=True`:**
- The tool definition is NOT included into the beginning of Claude\'s
context window.
- Instead, it\'s loaded into context when Claude sees the
`tool_reference`
- This means your system prompt and initial tools stay cached even as
you discover new tools
**The pattern:**
``` python
# Initial request - only describe_tool, system prompt is cached
tools = [DESCRIBE_TOOL]
# After Claude calls describe_tool("get_weather")
# Add with defer_loading to preserve cache
tools.append({**TOOL_LIBRARY["get_weather"], "defer_loading": True})
# Return tool_reference so Claude knows it's available
tool_result = [{"type": "tool_reference", "tool_name": "get_weather"}]
```
This is essential for applications with hundreds or thousands of tools
where you want to:
- Keep initial request size small
- Preserve prompt caching across tool discoveries
- Only load tools Claude actually needs
:::
::: {.cell .markdown}
## Conclusion
The key insight from this cookbook: **tools don\'t need to be in the
`tools` list until Claude needs them**. Combined with
`defer_loading=True` and `tool_reference`, this lets you scale to
thousands of tools while keeping requests small and preserving prompt
caching. However, in this case your client will need to provide a tool
discovery mechanism.
The `describe_tool` approach shown here is just one flavor. Other
patterns include:
- **`list_tools`** - Returns tool names matching a category or keyword
- **Hierarchical discovery** - Browse tool categories, then load
specific tools
- **Hybrid** - Combine listing with semantic search for large catalogs
The core pattern is always:
1. Return `tool_reference` when a tool is discovered
2. Add the tool with `defer_loading=True` to preserve caching
3. Claude can then use the tool immediately
See [Tool Search with Embeddings](https://github.com/anthropics/claude-cookbooks/blob/main/tool_use/tool_search_with_embeddings.ipynb)
for the embeddings-based approach.
:::
---
## Tool Search with Embeddings: Scaling Claude to Thousands of Tools
Source: https://github.com/anthropics/claude-cookbooks/blob/main/tool_use/tool_search_with_embeddings.ipynb
Path: 11-RAG-Search/claude-cookbooks-tool-use-tool-search-with-embeddings.md
# Tool Search with Embeddings: Scaling Claude to Thousands of Tools
Building Claude applications with dozens of specialized tools quickly
hits a wall: providing all tool definitions upfront consumes your
context window, increases latency and costs, and makes it harder for
Claude to find the right tool. Beyond \~100 tools, this approach becomes
impractical.
Semantic tool search solves this by treating tools as discoverable
resources. Instead of front-loading hundreds of definitions, you give
Claude a single `tool_search` tool that returns relevant capabilities on
demand, cutting context usage by 90%+ while enabling applications that
scale to thousands of tools.
**By the end of this cookbook, you\'ll be able to:**
- Implement client-side tool search to scale Claude applications from
dozens to thousands of tools
- Use semantic embeddings to dynamically discover relevant tools based
on task context
- Apply this pattern to domain-specific tool libraries (APIs, databases,
internal systems)
This pattern is used in production by teams managing large tool
ecosystems where context efficiency is critical. While we\'ll
demonstrate with a small set of tools for clarity, the same approach
scales seamlessly to libraries with hundreds or thousands of tools.
## Prerequisites
Before following this guide, ensure you have:
**Required Knowledge**
- Python fundamentals - comfortable with functions, dictionaries, and
basic data structures
- Basic understanding of Claude tool use - we recommend reading the
[Tool Use
Guide](https://docs.anthropic.com/en/docs/build-with-claude/tool-use)
first
**Required Tools**
- Python 3.11 or higher
- Anthropic API key ([get one
here](https://docs.anthropic.com/claude/reference/getting-started-with-the-api))
## Setup
First, install the required dependencies:
:::
::::: {.cell .code execution_count="68"}
``` python
# Note: we use -q to avoid printing too much to stdout
# Use --only-binary to avoid build issues with pythran
%pip install --only-binary :all: -q anthropic sentence-transformers numpy python-dotenv
```
::: {.output .stream .stderr}
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
:::
::: {.output .stream .stdout}
Note: you may need to restart the kernel to use updated packages.
:::
:::::
::: {.cell .markdown}
Ensure your `.env` file contains:
ANTHROPIC_API_KEY=your_key_here
Load your environment variables and configure the client:
:::
:::: {.cell .code execution_count="69"}
``` python
import json
import random
from datetime import datetime, timedelta
from typing import Any
import anthropic
import numpy as np
from dotenv import load_dotenv
from sentence_transformers import SentenceTransformer
# Load environment variables from .env file
load_dotenv()
# Define model constant for easy updates
MODEL = "claude-sonnet-4-5-20250929"
# Initialize Claude client (API key loaded from environment)
claude_client = anthropic.Anthropic()
# Load the SentenceTransformer model
# all-MiniLM-L6-v2 is a lightweight model with 384 dimensional embeddings
# It will be downloaded from HuggingFace on first use
print("Loading SentenceTransformer model...")
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
print("✓ Clients initialized successfully")
```
::: {.output .stream .stdout}
Loading SentenceTransformer model...
✓ Clients initialized successfully
:::
::::
::: {.cell .markdown}
## Define Tool Library
Before we can implement semantic search, we need tools to search
through. We\'ll create a library of 8 tools across two categories:
Weather and Finance.
In production applications, you might manage hundreds or thousands of
tools across your internal APIs, database operations, or third-party
integrations. The semantic search approach scales to these larger
libraries without modification - we\'re using a small set here purely
for demonstration clarity.
:::
:::: {.cell .code execution_count="70"}
``` python
# Define our tool library with 2 domains
TOOL_LIBRARY = [
# Weather Tools
{
"name": "get_weather",
"description": "Get the current weather in a given location",
"input_schema": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The unit of temperature",
},
},
"required": ["location"],
},
},
{
"name": "get_forecast",
"description": "Get the weather forecast for multiple days ahead",
"input_schema": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state",
},
"days": {
"type": "number",
"description": "Number of days to forecast (1-10)",
},
},
"required": ["location", "days"],
},
},
{
"name": "get_timezone",
"description": "Get the current timezone and time for a location",
"input_schema": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City name or timezone identifier",
}
},
"required": ["location"],
},
},
{
"name": "get_air_quality",
"description": "Get current air quality index and pollutant levels for a location",
"input_schema": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City name or coordinates",
}
},
"required": ["location"],
},
},
# Finance Tools
{
"name": "get_stock_price",
"description": "Get the current stock price and market data for a given ticker symbol",
"input_schema": {
"type": "object",
"properties": {
"ticker": {
"type": "string",
"description": "Stock ticker symbol (e.g., AAPL, GOOGL)",
},
"include_history": {
"type": "boolean",
"description": "Include historical data",
},
},
"required": ["ticker"],
},
},
{
"name": "convert_currency",
"description": "Convert an amount from one currency to another using current exchange rates",
"input_schema": {
"type": "object",
"properties": {
"amount": {
"type": "number",
"description": "Amount to convert",
},
"from_currency": {
"type": "string",
"description": "Source currency code (e.g., USD)",
},
"to_currency": {
"type": "string",
"description": "Target currency code (e.g., EUR)",
},
},
"required": ["amount", "from_currency", "to_currency"],
},
},
{
"name": "calculate_compound_interest",
"description": "Calculate compound interest for investments over time",
"input_schema": {
"type": "object",
"properties": {
"principal": {
"type": "number",
"description": "Initial investment amount",
},
"rate": {
"type": "number",
"description": "Annual interest rate (as percentage)",
},
"years": {"type": "number", "description": "Number of years"},
"frequency": {
"type": "string",
"enum": ["daily", "monthly", "quarterly", "annually"],
"description": "Compounding frequency",
},
},
"required": ["principal", "rate", "years"],
},
},
{
"name": "get_market_news",
"description": "Get recent financial news and market updates for a specific company or sector",
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Company name, ticker symbol, or sector",
},
"limit": {
"type": "number",
"description": "Maximum number of news articles to return",
},
},
"required": ["query"],
},
},
]
print(f"✓ Defined {len(TOOL_LIBRARY)} tools in the library")
```
::: {.output .stream .stdout}
✓ Defined 8 tools in the library
:::
::::
::: {.cell .markdown}
## Create Tool Embeddings
Semantic search works by comparing the *meaning* of text, rather than
just searching for keywords. To enable this, we need to convert each
tool definition into an **embedding vector** that captures its semantic
meaning.
Since our tool definitions are structured JSON objects with names,
descriptions, and parameters, we first convert each tool into a
human-readable text representation, then generate embedding vectors
using SentenceTransformer\'s `all-MiniLM-L6-v2` model.
We picked this model because it is:
- **Lightweight and fast** (only 384 dimensions vs 768+ for larger
models)
- **Runs locally** without requiring API calls
- **Sufficient for tool search** (you can experiment with larger models
for better accuracy)
Let\'s start by creating a function that converts tool definitions into
searchable text:
:::
:::: {.cell .code execution_count="71"}
``` python
def tool_to_text(tool: dict[str, Any]) -> str:
"""
Convert a tool definition into a text representation for embedding.
Combines the tool name, description, and parameter information.
"""
text_parts = [
f"Tool: {tool['name']}",
f"Description: {tool['description']}",
]
# Add parameter information
if "input_schema" in tool and "properties" in tool["input_schema"]:
params = tool["input_schema"]["properties"]
param_descriptions = []
for param_name, param_info in params.items():
param_desc = param_info.get("description", "")
param_type = param_info.get("type", "")
param_descriptions.append(f"{param_name} ({param_type}): {param_desc}")
if param_descriptions:
text_parts.append("Parameters: " + ", ".join(param_descriptions))
return "\n".join(text_parts)
# Test with one tool
sample_text = tool_to_text(TOOL_LIBRARY[0])
print("Sample tool text representation:")
print(sample_text)
```
::: {.output .stream .stdout}
Sample tool text representation:
Tool: get_weather
Description: Get the current weather in a given location
Parameters: location (string): The city and state, e.g. San Francisco, CA, unit (string): The unit of temperature
:::
::::
::: {.cell .markdown}
Now let\'s create embeddings for all our tools:
:::
:::: {.cell .code execution_count="72"}
``` python
# Create embeddings for all tools
print("Creating embeddings for all tools...")
tool_texts = [tool_to_text(tool) for tool in TOOL_LIBRARY]
# Embed all tools at once using SentenceTransformer
# The model returns normalized embeddings by default
tool_embeddings = embedding_model.encode(tool_texts, convert_to_numpy=True)
print(f"✓ Created embeddings with shape: {tool_embeddings.shape}")
print(f" - {tool_embeddings.shape[0]} tools")
print(f" - {tool_embeddings.shape[1]} dimensions per embedding")
```
::: {.output .stream .stdout}
Creating embeddings for all tools...
✓ Created embeddings with shape: (8, 384)
- 8 tools
- 384 dimensions per embedding
:::
::::
::: {.cell .markdown}
## Implement Tool Search
With our tools embedded as vectors, we can now implement semantic
search. If two pieces of text have similar meanings, their embedding
vectors will be close together in vector space. We measure this
\"closeness\" using **cosine similarity**.
The search process:
1. **Embed the query**: Convert Claude\'s natural language search
request into the same vector space as our tools
2. **Calculate similarity**: Compute cosine similarity between the
query vector and each tool vector
3. **Rank and return**: Sort tools by similarity score and return the
top N matches
With semantic search, Claude can search using natural language like \"I
need to check the weather\" or \"calculate investment returns\" rather
than exact tool names.
Let\'s implement the search function and test it with a sample query:
:::
:::: {.cell .code execution_count="73"}
``` python
def search_tools(query: str, top_k: int = 5) -> list[dict[str, Any]]:
"""
Search for tools using semantic similarity.
Args:
query: Natural language description of what tool is needed
top_k: Number of top tools to return
Returns:
List of tool definitions most relevant to the query
"""
# Embed the query using SentenceTransformer
query_embedding = embedding_model.encode(query, convert_to_numpy=True)
# Calculate cosine similarity using dot product
# SentenceTransformer returns normalized embeddings, so dot product = cosine similarity
similarities = np.dot(tool_embeddings, query_embedding)
# Get top k indices
top_indices = np.argsort(similarities)[-top_k:][::-1]
# Return the corresponding tools with their scores
results = []
for idx in top_indices:
results.append({"tool": TOOL_LIBRARY[idx], "similarity_score": float(similarities[idx])})
return results
# Test the search function
test_query = "I need to check the weather"
test_results = search_tools(test_query, top_k=3)
print(f"Search query: '{test_query}'\n")
print("Top 3 matching tools:")
for i, result in enumerate(test_results, 1):
tool_name = result["tool"]["name"]
score = result["similarity_score"]
print(f"{i}. {tool_name} (similarity: {score:.3f})")
```
::: {.output .stream .stdout}
Search query: 'I need to check the weather'
Top 3 matching tools:
1. get_weather (similarity: 0.560)
2. get_forecast (similarity: 0.508)
3. get_air_quality (similarity: 0.401)
:::
::::
::: {.cell .markdown}
## Define the tool_search Tool
Now we\'ll implement the **meta-tool** that allows Claude to discover
other tools on demand. When Claude needs a capability it doesn\'t have,
it searches for it using this `tool_search` tool, receives the tool
definitions in the result, and can use those newly discovered tools
immediately.
This is the only tool we provide to Claude initially:
:::
:::: {.cell .code execution_count="74"}
``` python
# The tool_search tool definition
TOOL_SEARCH_DEFINITION = {
"name": "tool_search",
"description": "Search for available tools that can help with a task. Returns tool definitions for matching tools. Use this when you need a tool but don't have it available yet.",
"input_schema": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Natural language description of what kind of tool you need (e.g., 'weather information', 'currency conversion', 'stock prices')",
},
"top_k": {
"type": "number",
"description": "Number of tools to return (default: 5)",
},
},
"required": ["query"],
},
}
print("✓ Tool search definition created")
```
::: {.output .stream .stdout}
✓ Tool search definition created
:::
::::
::: {.cell .markdown}
Now let\'s implement the handler that processes `tool_search` calls from
Claude and returns discovered tools:
:::
:::: {.cell .code execution_count="75"}
``` python
def handle_tool_search(query: str, top_k: int = 5) -> list[dict[str, Any]]:
"""
Handle a tool_search invocation and return tool references.
Returns a list of tool_reference content blocks for discovered tools.
"""
# Search for relevant tools
results = search_tools(query, top_k=top_k)
# Create tool_reference objects instead of full definitions
tool_references = [
{"type": "tool_reference", "tool_name": result["tool"]["name"]} for result in results
]
print(f"\n🔍 Tool search: '{query}'")
print(f" Found {len(tool_references)} tools:")
for i, result in enumerate(results, 1):
print(f" {i}. {result['tool']['name']} (similarity: {result['similarity_score']:.3f})")
return tool_references
# Test the handler
test_result = handle_tool_search("stock market data", top_k=3)
print(f"\nReturned {len(test_result)} tool references:")
for ref in test_result:
print(f" {ref}")
```
::: {.output .stream .stdout}
🔍 Tool search: 'stock market data'
Found 3 tools:
1. get_stock_price (similarity: 0.524)
2. get_market_news (similarity: 0.469)
3. calculate_compound_interest (similarity: 0.244)
Returned 3 tool references:
{'type': 'tool_reference', 'tool_name': 'get_stock_price'}
{'type': 'tool_reference', 'tool_name': 'get_market_news'}
{'type': 'tool_reference', 'tool_name': 'calculate_compound_interest'}
:::
::::
::: {.cell .markdown}
## Mock Tool Execution
For this demonstration, we\'ll create mock responses for tool
executions. In a real application, these would call actual APIs or
services:
:::
:::: {.cell .code execution_count="76"}
``` python
def mock_tool_execution(tool_name: str, tool_input: dict[str, Any]) -> str:
"""
Generate realistic mock responses for tool executions.
Args:
tool_name: Name of the tool being executed
tool_input: Input parameters for the tool
Returns:
Mock response string appropriate for the tool
"""
# Weather tools
if tool_name == "get_weather":
location = tool_input.get("location", "Unknown")
unit = tool_input.get("unit", "fahrenheit")
temp = random.randint(15, 30) if unit == "celsius" else random.randint(60, 85)
conditions = random.choice(["sunny", "partly cloudy", "cloudy", "rainy"])
return json.dumps(
{
"location": location,
"temperature": temp,
"unit": unit,
"conditions": conditions,
"humidity": random.randint(40, 80),
"wind_speed": random.randint(5, 20),
}
)
elif tool_name == "get_forecast":
location = tool_input.get("location", "Unknown")
days = int(tool_input.get("days", 5))
forecast = []
for i in range(days):
date = (datetime.now() + timedelta(days=i)).strftime("%Y-%m-%d")
forecast.append(
{
"date": date,
"high": random.randint(20, 30),
"low": random.randint(10, 20),
"conditions": random.choice(["sunny", "cloudy", "rainy", "partly cloudy"]),
}
)
return json.dumps({"location": location, "forecast": forecast})
elif tool_name == "get_timezone":
location = tool_input.get("location", "Unknown")
return json.dumps(
{
"location": location,
"timezone": "UTC+9",
"current_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"utc_offset": "+09:00",
}
)
elif tool_name == "get_air_quality":
location = tool_input.get("location", "Unknown")
aqi = random.randint(20, 150)
categories = {
(0, 50): "Good",
(51, 100): "Moderate",
(101, 150): "Unhealthy for Sensitive Groups",
}
category = next(cat for (low, high), cat in categories.items() if low <= aqi <= high)
return json.dumps(
{
"location": location,
"aqi": aqi,
"category": category,
"pollutants": {
"pm25": random.randint(5, 50),
"pm10": random.randint(10, 100),
"o3": random.randint(20, 80),
},
}
)
# Finance tools
elif tool_name == "get_stock_price":
ticker = tool_input.get("ticker", "UNKNOWN")
return json.dumps(
{
"ticker": ticker,
"price": round(random.uniform(100, 500), 2),
"change": round(random.uniform(-5, 5), 2),
"change_percent": round(random.uniform(-2, 2), 2),
"volume": random.randint(1000000, 10000000),
"market_cap": f"${random.randint(100, 1000)}B",
}
)
elif tool_name == "convert_currency":
amount = tool_input.get("amount", 0)
from_currency = tool_input.get("from_currency", "USD")
to_currency = tool_input.get("to_currency", "EUR")
# Mock exchange rate
rate = random.uniform(0.8, 1.2)
converted = round(amount * rate, 2)
return json.dumps(
{
"original_amount": amount,
"from_currency": from_currency,
"to_currency": to_currency,
"exchange_rate": round(rate, 4),
"converted_amount": converted,
}
)
elif tool_name == "calculate_compound_interest":
principal = tool_input.get("principal", 0)
rate = tool_input.get("rate", 0)
years = tool_input.get("years", 0)
frequency = tool_input.get("frequency", "monthly")
# Calculate compound interest
n_map = {"daily": 365, "monthly": 12, "quarterly": 4, "annually": 1}
n = n_map.get(frequency, 12)
final_amount = principal * (1 + rate / 100 / n) ** (n * years)
interest_earned = final_amount - principal
return json.dumps(
{
"principal": principal,
"rate": rate,
"years": years,
"compounding_frequency": frequency,
"final_amount": round(final_amount, 2),
"interest_earned": round(interest_earned, 2),
}
)
elif tool_name == "get_market_news":
query = tool_input.get("query", "")
limit = tool_input.get("limit", 5)
news = []
for i in range(min(limit, 5)):
news.append(
{
"title": f"{query} - News Article {i + 1}",
"source": random.choice(
[
"Bloomberg",
"Reuters",
"Financial Times",
"Wall Street Journal",
]
),
"published": (datetime.now() - timedelta(hours=random.randint(1, 24))).strftime(
"%Y-%m-%d %H:%M"
),
"summary": f"Latest developments regarding {query}...",
}
)
return json.dumps({"query": query, "articles": news, "count": len(news)})
# Default fallback
else:
return json.dumps(
{
"status": "executed",
"tool": tool_name,
"message": f"Tool {tool_name} executed successfully with input: {json.dumps(tool_input)}",
}
)
print("✓ Mock tool execution function created")
```
::: {.output .stream .stdout}
✓ Mock tool execution function created
:::
::::
::: {.cell .markdown}
## Implement Conversation Loop
Now let\'s put it all together! We\'ll create a conversation loop that
handles the complete tool search workflow.
**The conversation flow:**
1. Claude starts with only the `tool_search` tool available
2. When Claude calls `tool_search`, we run semantic search and return
matching tool definitions
3. Claude can then use the discovered tools immediately
4. When Claude calls a discovered tool, we execute it (using mock
responses for this demo)
5. The loop continues until Claude has a final answer
:::
:::: {.cell .code execution_count="77"}
``` python
def run_tool_search_conversation(user_message: str, max_turns: int = 5) -> None:
"""
Run a conversation with Claude using the tool search pattern.
Args:
user_message: The initial user message
max_turns: Maximum number of conversation turns
"""
print(f"\n{'=' * 80}")
print(f"USER: {user_message}")
print(f"{'=' * 80}\n")
# Initialize conversation with only tool_search available
messages = [{"role": "user", "content": user_message}]
for turn in range(max_turns):
print(f"\n--- Turn {turn + 1} ---")
# Call Claude with current message history
response = claude_client.messages.create(
model=MODEL,
max_tokens=1024,
tools=TOOL_LIBRARY + [TOOL_SEARCH_DEFINITION],
messages=messages,
# IMPORTANT: This beta header enables tool definitions in tool results
extra_headers={"anthropic-beta": "advanced-tool-use-2025-11-20"},
)
# Add assistant's response to messages
messages.append({"role": "assistant", "content": response.content})
# Check if we're done
if response.stop_reason == "end_turn":
print("\n✓ Conversation complete\n")
# Print final response
for block in response.content:
if block.type == "text":
print(f"ASSISTANT: {block.text}")
break
# Handle tool uses
if response.stop_reason == "tool_use":
tool_results = []
for block in response.content:
if block.type == "text":
print(f"\nASSISTANT: {block.text}")
elif block.type == "tool_use":
tool_name = block.name
tool_input = block.input
tool_use_id = block.id
print(f"\n🔧 Tool invocation: {tool_name}")
print(f" Input: {json.dumps(tool_input, indent=2)}")
if tool_name == "tool_search":
# Handle tool search
query = tool_input["query"]
top_k = tool_input.get("top_k", 5)
# Get tool references
tool_references = handle_tool_search(query, top_k)
# Create tool result with tool_reference content blocks
tool_results.append(
{
"type": "tool_result",
"tool_use_id": tool_use_id,
"content": tool_references,
}
)
else:
# Execute the discovered tool with mock data
mock_result = mock_tool_execution(tool_name, tool_input)
# Print a preview of the result
if len(mock_result) > 150:
print(f" ✅ Mock result: {mock_result[:150]}...")
else:
print(f" ✅ Mock result: {mock_result}")
tool_results.append(
{
"type": "tool_result",
"tool_use_id": tool_use_id,
"content": mock_result,
}
)
# Add tool results to messages
if tool_results:
messages.append({"role": "user", "content": tool_results})
else:
print(f"\nUnexpected stop reason: {response.stop_reason}")
break
print(f"\n{'=' * 80}\n")
print("✓ Conversation loop implemented")
```
::: {.output .stream .stdout}
✓ Conversation loop implemented
:::
::::
::: {.cell .markdown}
## Example 1: Weather Query
Let\'s test with a simple weather question. Claude should:
1. Call `tool_search` to find weather tools
2. Receive weather tool definitions in the result
3. Use one of the discovered tools
:::
:::: {.cell .code execution_count="78"}
``` python
run_tool_search_conversation("What's the weather like in Tokyo?")
```
::: {.output .stream .stdout}
================================================================================
USER: What's the weather like in Tokyo?
================================================================================
--- Turn 1 ---
🔧 Tool invocation: get_weather
Input: {
"location": "Tokyo"
}
✅ Mock result: {"location": "Tokyo", "temperature": 75, "unit": "fahrenheit", "conditions": "partly cloudy", "humidity": 61, "wind_speed": 9}
--- Turn 2 ---
✓ Conversation complete
ASSISTANT: The weather in Tokyo is currently:
- **Temperature:** 75°F (about 24°C)
- **Conditions:** Partly cloudy
- **Humidity:** 61%
- **Wind Speed:** 9 mph
It's a pleasant day with comfortable temperatures and some cloud cover!
================================================================================
:::
::::
::: {.cell .markdown}
## Example 2: Finance Query
Let\'s try a financial calculation query that requires discovering and
using finance tools:
:::
:::: {.cell .code execution_count="79"}
``` python
run_tool_search_conversation(
"If I invest $10,000 at 5% annual interest for 10 years with monthly compounding, how much will I have?"
)
```
::: {.output .stream .stdout}
================================================================================
USER: If I invest $10,000 at 5% annual interest for 10 years with monthly compounding, how much will I have?
================================================================================
--- Turn 1 ---
🔧 Tool invocation: calculate_compound_interest
Input: {
"principal": 10000,
"rate": 5,
"years": 10,
"frequency": "monthly"
}
✅ Mock result: {"principal": 10000, "rate": 5, "years": 10, "compounding_frequency": "monthly", "final_amount": 16470.09, "interest_earned": 6470.09}
--- Turn 2 ---
✓ Conversation complete
ASSISTANT: If you invest $10,000 at 5% annual interest for 10 years with monthly compounding, you will have:
**Final Amount: $16,470.09**
This means you'll earn **$6,470.09** in interest over the 10-year period.
The monthly compounding means that interest is calculated and added to your principal every month, which allows your investment to grow faster than with annual compounding due to the effect of earning "interest on interest" more frequently.
================================================================================
:::
::::
::: {.cell .markdown}
## Conclusion
In this cookbook, we implemented a client-side tool search system that
enables Claude to work with large tool libraries efficiently. We
covered:
- **Semantic tool discovery**: Using embeddings to match natural
language queries to relevant tools, enabling Claude to find the right
capability without seeing all available tools upfront
- **Dynamic tool loading**: Returning tool definitions in tool results
using Claude\'s tool search feature, allowing Claude to discover and
immediately use new tools mid-conversation
- **Context optimization**: Reducing initial context from thousands of
tokens (19+ tool definitions) to just the single `tool_search`
definition, cutting context usage by 90%+
### Applying This to Your Projects
Consider tool search when:
- You have **\>20 specialized tools** and context usage becomes a
concern
- Your tool library **grows over time** and manual curation becomes
impractical
- You need to support **domain-specific APIs** with hundreds of
endpoints (database operations, internal microservices, third-party
integrations)
- **Cost and latency optimization** are priorities for your application
### Next Steps
To take this implementation further:
1. **Persist embeddings**: Cache embeddings to disk to avoid
recomputing on every session, reducing startup time
2. **Improve search quality**: Experiment with different embedding
models (e.g., larger models like `all-mpnet-base-v2`) or implement
hybrid search combining semantic and keyword matching (BM25)
3. **Scale to larger libraries**: Test with hundreds or thousands of
tools to see how the pattern performs at production scale
4. **Add tool metadata**: Include usage statistics, cost information,
or reliability scores in your search ranking
5. **Implement caching**: Cache frequently used tool definitions to
reduce repeated searches
### Further Reading
- [Claude Tool Use
Guide](https://docs.anthropic.com/en/docs/build-with-claude/tool-use) -
Comprehensive guide to building with tools
- [SentenceTransformers Documentation](https://www.sbert.net/) - Learn
more about embedding models and semantic search
- [Tool Search Tool
Documentation](https://docs.anthropic.com/en/docs/build-with-claude/tool-use#tool-search) -
Official documentation on the tool search pattern
:::