langfuse-tracing · diff
git:20260724.d4db756 to git:20260724.26faef0
103 added, 52 removed. Audit A to A.
---
name: langfuse-tracing
- description: "Wires Langfuse tracing into LLM apps for production observability and offline eval - instruments via `@observe` (Python) / `startActiveObservation` (TS) decorators that auto-capture inputs / outputs / timings / errors per generation; exposes `langfuse.update_current_span()` for metadata + cost / latency annotation; supports trace-bound scoring for eval datasets and prompt-as-code management. Use when the user needs production LLM observability beyond pre-deploy eval, or wants to ship traces from production to an eval dataset for offline regression testing."
+ description: "Wires Langfuse tracing into LLM apps for production observability, monitoring, telemetry, and offline eval - instruments via `@observe` (Python) / `startActiveObservation` (TS) decorators that auto-capture inputs / outputs / timings / errors per generation; exposes `langfuse.update_current_span()` for metadata + cost / latency annotation; supports trace-bound scoring for eval datasets and prompt-as-code management. Use when the user needs to monitor, log, trace, or debug LLM API calls in production beyond pre-deploy eval, wants to add LLM observability tooling to an existing app, or wants to ship traces from production to an eval dataset for offline regression testing."
---
# langfuse-tracing
## Overview
[lf-gh]: https://github.com/langfuse/langfuse-python
- Langfuse complements pre-deploy LLM eval (Promptfoo / DeepEval /
- Ragas / Giskard) with production-side observability - captures every
- LLM call as a `trace` containing nested `observation`s (generations,
- spans, events), with token / cost / latency metadata, scores, and
- linked datasets for offline eval (per [lf-gh][lf-gh]).
+ Langfuse is a production LLM observability platform (per [lf-gh][lf-gh]).
**Important version note (2026-05-06):** per [lf-gh][lf-gh], "The
SDK was rewritten in v4 and released in March 2026" - this skill
targets the v4 API. For v3 codebases, see the upstream migration
guide.
- ## When to use
-
- - An LLM app is in production and the team needs observability:
- per-call traces, cost/latency dashboards, error tracking.
- - The team wants to capture production traces into an eval dataset
- for offline regression testing.
- - Prompt-as-code management is needed (version pin prompts, A/B
- test prompt variants in production).
- - An eval framework (Promptfoo / DeepEval / Ragas / Giskard) is
- in place pre-deploy; Langfuse extends it post-deploy.
-
## Step 1 - Install
Per [lf-gh][lf-gh]:
```bash
pip install langfuse
```
For TypeScript:
```bash
npm install @langfuse/tracing
```
Set up project credentials per Langfuse self-hosted or cloud
project (`LANGFUSE_PUBLIC_KEY` + `LANGFUSE_SECRET_KEY` +
`LANGFUSE_HOST`).
## Step 2 - Instrument with `@observe`
Per [langfuse.com/docs/sdk/python/decorators][lf-py-deco]:
[lf-py-deco]: https://langfuse.com/docs/sdk/python/decorators
Python:
```python
from langfuse import observe
- @observe(name="llm-call", as_type="generation")
+ @observe(name="llm-call", as_type="generation") # auto-captures inputs, outputs, timings, errors
async def my_async_llm_call(prompt_text):
return "LLM response"
```
- The decorator "automatically captures inputs, outputs, timings, and
- errors without modifying function logic" (per [lf-py-deco][lf-py-deco]).
-
- TypeScript:
+ TypeScript - create an observation, do work inside it, then end it:
```typescript
- import { startActiveObservation, startObservation } from "@langfuse/tracing";
+ import { startActiveObservation } from "@langfuse/tracing";
+
+ const { observation, end } = startActiveObservation({
+ name: "llm-call",
+ type: "generation",
+ input: { prompt: promptText },
+ });
+
+ try {
+ const result = await callMyLLM(promptText);
+ observation.update({ output: result });
+ return result;
+ } finally {
+ end();
+ }
```
- (Per [lf-py-deco][lf-py-deco]; the TS SDK uses an explicit
- `startObservation` API rather than a decorator.)
+ **Validation:** After your first instrumented call, open the Langfuse
+ UI → Traces. You should see a new trace with nested observations,
+ inputs, outputs, and timings. If no trace appears within ~30 s:
+ - Confirm `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, and
+ `LANGFUSE_HOST` are set in the process environment.
+ - Call `await langfuse.flushAsync()` (TS) / `langfuse.flush()` (Python)
+ before process exit to force-drain the queue.
+ - Check `langfuse.debug()` output for connection errors.
## Step 3 - Update current observation with metadata
Per [lf-py-deco][lf-py-deco]:
```python
from langfuse import get_client
langfuse = get_client()
with langfuse.start_as_current_observation(as_type="span", name="data-processing"):
langfuse.update_current_span(metadata={"step1_complete": True})
```
Common metadata fields used in production:
- `model` - the model name (e.g., `claude-haiku-4-5`)
- `model_parameters` - temperature / top_p / max_tokens
- `usage` - input/output tokens, cost
- `tags` - environment (`prod` / `staging`), feature flag, customer ID
- `level` - `DEBUG` / `DEFAULT` / `WARNING` / `ERROR`
## Step 4 - Score traces
- Scores attach evaluation results to a trace or observation. Three
- data types per Langfuse: numeric (0 - 1), categorical (string), boolean
- (true/false). The full API and example invocation live at
- [langfuse.com/docs/scores][lf-scores]; the trace-side wiring is:
+ Per [langfuse.com/docs/scores][lf-scores]:
[lf-scores]: https://langfuse.com/docs/scores
```python
langfuse.score(
trace_id="...",
name="answer_relevance",
- value=0.87, # numeric
+ value=0.87, # numeric (0-1); also supports categorical (string) and boolean
comment="Judged by GPT-4 rubric"
)
```
- Per [lf-scores][lf-scores] the current Python SDK API is the source
- of truth - consult that page when wiring scores. Scores can come from:
+ Scores can come from:
- **Manual review** (human raters via the Langfuse UI)
- **Automated eval** (run a metric in batch, score by `trace_id`)
- **User feedback** (thumbs-up / thumbs-down from the app)
+ **Validation:** After scoring, open the Langfuse UI → Traces →
+ select the trace. The score should appear in the Scores panel.
+ If missing, confirm the `trace_id` matches an existing trace and
+ that the score call did not raise an exception.
+
## Step 5 - Datasets for offline eval
Langfuse datasets (collections of `(input, expected_output)` items)
- can be:
-
- - Built from production traces (capture inputs that produced bad
- scores; promote them into a regression dataset)
- - Pulled from CSV / JSONL imports
- - Hand-authored via the Langfuse UI
+ can be built from production traces, CSV / JSONL imports, or the UI.
Run a dataset:
```python
items = langfuse.get_dataset_items(dataset_id="...")
for item in items:
actual = my_llm_app(item.input)
item.run(actual) # links the run back to the dataset for diff vs baseline
```
- (API exact signature evolves; see [langfuse.com/docs/datasets][lf-ds].)
+ (See [langfuse.com/docs/datasets][lf-ds] for the current API signature.)
[lf-ds]: https://langfuse.com/docs/datasets
+ **Validation:** After running a dataset, open the Langfuse UI →
+ Datasets → select your dataset. Each item run should appear under
+ the Runs tab linked to its trace. If runs are missing, confirm
+ `dataset_id` is correct and that `item.run()` did not raise an
+ exception.
+
## Step 6 - Prompt management
- Pin prompt versions in code; iterate prompts in the Langfuse UI;
- roll out new prompt versions per environment (`production` /
- `staging` labels) without code deploys. The `langfuse.get_prompt()`
- API fetches the current production prompt at runtime.
+ Fetch the current production prompt at runtime and pin its version
+ so prompt drift is traceable:
+ ```python
+ from langfuse import get_client
+
+ langfuse = get_client()
+
+ # Fetch the production-labelled prompt; falls back to cache if offline
+ prompt = langfuse.get_prompt("my-prompt-name", label="production")
+ compiled = prompt.compile(user_input=user_query)
+
+ # Pass compiled text to your LLM call
+ response = my_llm_call(compiled)
+ ```
+
+ Iterate prompt text in the Langfuse UI and roll out new versions
+ per environment (`production` / `staging` labels) without code
+ deploys. Version history and A/B comparison are available in the UI.
+
+ **Validation:** After an instrumented call using a managed prompt,
+ open the Langfuse UI → Traces → select the trace. The prompt name
+ and version should appear in the trace metadata, confirming prompt
+ version is pinned and attributable.
+
## Step 7 - CI integration
- Langfuse is observability-side, not pre-deploy CI-side. CI integration
- patterns:
+ Langfuse is observability-side, not pre-deploy CI-side. Typical
+ post-deploy CI patterns use the Langfuse API to query recent traces
+ and assert on aggregate metrics:
+ ```python
+ import httpx, os, sys
+ from datetime import datetime, timedelta, timezone
+
+ LANGFUSE_HOST = os.environ["LANGFUSE_HOST"]
+ headers = {"Authorization": f"Bearer {os.environ['LANGFUSE_SECRET_KEY']}"}
+
+ # Fetch average answer_relevance score over the last hour
+ one_hour_ago = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()
+ resp = httpx.get(
+ f"{LANGFUSE_HOST}/api/public/scores",
+ params={"name": "answer_relevance", "fromTimestamp": one_hour_ago},
+ headers=headers,
+ )
+ scores = resp.json()["data"]
+ avg = sum(s["value"] for s in scores) / len(scores) if scores else 1.0
+
+ if avg < 0.75:
+ print(f"answer_relevance regression: {avg:.2f} < 0.75 threshold")
+ sys.exit(1)
+ ```
+
+ **Validation:** After the CI job runs, confirm the script exits 0
+ and that the queried score window covers the expected deployment
+ window. If `scores` is empty, verify the `fromTimestamp` range and
+ that instrumented calls have been scored in that period.
+
+ Other CI wiring patterns:
- **Eval-on-trace**: post-deploy, run an offline eval sweep against
production traces from the previous N hours; fail if regression.
- - **Score-based alerting**: alert on drop in average `answer_relevancy`
- score over a rolling window.
- **Cost regression**: alert on per-trace cost increase that exceeds
budget.
-
- These are dashboard / alerting wires (Langfuse → PagerDuty / Slack /
- Datadog), not CI-pipeline assertions.
+ - **Score-based alerting**: route to PagerDuty / Slack / Datadog
+ on drop in average score over a rolling window.
## Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Trace everything in production with no sampling | Cost explodes at scale | Use `level=DEBUG` + UI-side sampling (Step 3) |
| Score traces only via UI (no automated path) | Can't catch silent regressions | Automated `langfuse.score()` per trace (Step 4) |
| Pull production trace inputs without privacy review | PII leakage into eval datasets | Cross-ref `synthetic-pii-generator` for fixture sanitization before promotion |
| Skip prompt versioning | Prompt drift breaks attribution | `langfuse.get_prompt()` with version pin (Step 6) |
| Conflate Langfuse with pre-deploy eval | Tries to be both; wins neither | Pair Langfuse (post-deploy) with Promptfoo/DeepEval/Ragas (pre-deploy) |
## Limitations
- Langfuse cloud is hosted; for data-residency-strict teams,
self-host (well-supported but operational overhead).
- v4 API rewrite (March 2026) - pin SDK version in requirements;
v3 patterns no longer supported per [lf-gh][lf-gh].
+ - SDK and score-API signatures may drift between versions; always
+ consult [langfuse.com/docs][lf-docs] and [lf-scores][lf-scores]
+ when authoring wiring against any rapidly-evolving endpoint.
- Integration with eval frameworks evolves rapidly; consult
[langfuse.com/docs][lf-docs] for current Promptfoo / DeepEval /
Ragas integration patterns.
- - Score-API exact signature may drift between SDK versions;
- always check [lf-scores][lf-scores] when authoring score wiring.
[lf-docs]: https://langfuse.com/docs
## References
- [lf-gh][lf-gh] - Python SDK repo, install, version note
- [lf-py-deco][lf-py-deco] - `@observe` decorator + observation
patterns
- [lf-scores][lf-scores] - score API
- [lf-ds][lf-ds] - datasets
- [lf-docs][lf-docs] - full documentation
- `promptfoo-evaluation`,
`deepeval-evaluation`,
`ragas-evaluation`,
`giskard-llm` - pre-deploy eval sister
tools