Stop Playing Tebak-Tebak Berhadiah With Your AI Agent

Someone builds an AI agent. It reads a customer's question, calls a couple of tools, searches a knowledge base, and writes an answer. They try it on five or six questions they thought of, every answer looks good, and it goes live.

A week later a customer complains that the bot said their order had shipped when it hadn't. The developer adds a sentence to the prompt, "only report order status from the lookup tool", reruns the same five questions, and they still pass. Ship it.

Did that fix it? Did it break something else? How often was it happening in the first place? Nobody knows. In Indonesia we call this tebak-tebak berhadiah: the guessing game at a 17 Agustus neighborhood party, where someone shouts an answer and hopes for a prize. It's a fine way to spend an afternoon in a kampung. As a release process, it's how you lose customers without ever learning why.

The fix is engineering, and most of it is unglamorous: counting, sampling, and storing the right data in the right shape. Langfuse happens to be a good place to see all of it, because its own architecture went through the same lessons.

Five passing prompts rule out almost nothing

Treat each run of your agent as a trial that comes out correct with some unknown probability. Your five test prompts are a sample of that distribution, and a sample of five is tiny.

Say the agent is wrong one time in five. The chance that all five of your tries pass anyway is 0.8⁵, about 33%. A third of the time, your test session reports a clean bill of health on an agent that fails 20% of real requests.

Now ask what a clean streak actually proves. With zero failures in n runs, the highest failure rate you can rule out at 95% confidence is 1 − 0.05^1/n:

The bottom rows follow the rule of three: zero failures in n runs puts the true rate below roughly 3/n. To claim your agent fails less than 5% of the time, you need about 60 clean runs on realistic inputs, which nobody types by hand.

Agents compound the problem because a run is a chain of decisions, and errors multiply along it. If each step (pick a tool, fill in its arguments, read the result correctly) is right 95% of the time, a 10-step run is fully right 0.95¹⁰, about 60% of the time.

Consistency matters as much as capability. The τ-bench paper [1] measures pass^k, the probability that an agent succeeds on all k attempts at the same task. GPT-4o solved under half of its tasks, and its pass⁸ in the retail domain was under 25%. A customer who asks the same thing twice and gets two different answers has just measured your pass².

Why your usual tooling doesn't see it

How Langfuse is built, and why it changes your instrumentation

Langfuse is an open-source platform for tracing and evaluating LLM applications. The main repository is MIT licensed apart from its enterprise folders, and ClickHouse acquired the company in January 2026 with no planned license changes [3]. For an engineer, the plumbing is the interesting part, because a few of its design decisions dictate how you should write your instrumentation.

The SDK is OpenTelemetry underneath

The Python and JS/TS SDKs are built on OpenTelemetry, and an observation is an OTel span. The @observe decorator opens a span and puts it in the OTel context (contextvars, in Python), so any observed function called inside it becomes a child automatically, through nested calls and async tasks. Finished spans are queued in memory and exported in batches by a background thread, which keeps tracing off your request path.

Batching has a price. A short-lived process can exit before the exporter runs: a serverless runtime may freeze the process right after the response, and a CLI or cron job simply ends. Call langfuse.flush() before that point or the last traces never arrive.

Ingestion is a queue, so reads lag writes

Langfuse v2 did all of its work inside the ingestion request. Each event meant looking up the existing row, tokenizing the prompt and completion, computing cost, and upserting into Postgres. Bursty batch traffic exhausted Postgres IOPS and burned CPU on tokenization, and ingestion latency spiked to 50 seconds in the summer of 2023 [4].

v3 split the path in two. The API checks auth and body format, writes the raw events to S3, and pushes a reference onto a Redis queue. A worker pulls the reference, loads every event for that object, computes the final state, and inserts it into ClickHouse without reading from ClickHouse first, which sidesteps read-after-write races. Raw events in S3 can be replayed after a failure, and Redis only carries pointers, which keeps it clear of its network limits.

The consequence for you is eventual consistency. A trace appears seconds after the request ends, so a test that writes a trace and immediately reads it back through the API can fail for reasons unrelated to your code.

v4 rebuilt storage around observations

In v3, traces and observations lived in separate ClickHouse tables on ReplacingMergeTree. An update was written as a new full row, duplicates only disappeared when background merges got around to them, and queries had to deduplicate at read time with FINAL or LIMIT BY. Any question like cost per user needed a join between the two tables. Data processed grew 19x over a year, and the largest projects could only load a few days of dashboard data [5].

v4 replaces this with one wide, mostly immutable observations table. OTel spans are immutable once they end, and roughly 60% of Langfuse Cloud observations already arrive over the OTel endpoint, so most rows never need a merge. trace_id becomes a correlation column, like session_id. Attributes such as user_id, session_id, and tags are written onto every observation at ingestion, so cost per user becomes a column filter plus a GROUP BY, with no join [5].

It's the same lesson as my Elasticsearch post-mortem: shape storage for the queries you actually run, and make every write carry what those queries need.

What that means for your code

• Set attributes at the top of the request. propagate_attributes applies user_id, session_id, and tags to the current observation and its children inside its scope [6]. Steps that start before you enter it won't carry them, and filters on those attributes won't find those steps.
• Upgrade the SDK. Real-time data on v4 needs Python SDK 4.7.0 or JS/TS SDK 5.4.0 and up, and older SDKs can lag by up to 15 minutes. Langfuse Cloud deprecates the v3 endpoints on November 16, 2026, and trace-level LLM judges must move to observations by then [7].
• Don't assert on fresh writes. Read your own traces back with a retry, or assert on the experiment result object your script already holds.

Instrumenting an agent

Three environment variables, the drop-in OpenAI client so every model call gets logged with model, tokens, and cost, and a decorator on your own functions so they show up as typed steps:

pip install "langfuse>=4.7"

export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"  # or your self-hosted URL

from langfuse import observe, propagate_attributes
from langfuse.openai import openai  # drop-in client, every model call is logged

@observe(as_type="tool")
def lookup_order(order_id: str) -> dict:
    return orders_api.get(order_id)

@observe(as_type="agent")
def support_agent(question: str, user_id: str, chat_id: str) -> str:
    # Enter this before any child step starts, so every observation carries it
    with propagate_attributes(user_id=user_id, session_id=chat_id):
        return run_agent_loop(question, tools=[lookup_order])

Use the observation types. Langfuse has ten, including agent, tool, retriever, generation, and guardrail [8], and a typed trace reads like your design instead of a column of anonymous spans. Since it's all OTel, a request that crosses services (an API that enqueues a job for a worker) can stay one trace if you pass the W3C trace context along with the job.

Harnesses need this as much as chatbots do. If you run agents inside sandboxes (I built Kotak Pasir for that), the sandbox decides what the agent is allowed to do. The trace records what it actually tried.

Evals: push each check down to the cheapest layer that can decide it

A score is a number, boolean, category, or text note attached to an observation. Where scores come from matters far more than the dashboard they land on. I sort evaluators by cost and variance, and move every check as far down that list as it will go.

Code evaluators: zero variance, almost zero cost

Anything code can decide should be decided by code. For agents, the most useful checks look at the trajectory as well as the final text:

• the lookup tool ran before the answer stated an order status
• tool arguments validate against the tool's schema
• the run finished inside its step budget
• no tool was called twice in a row with identical arguments, the usual sign of a stuck loop
• the output parses as the JSON your frontend expects

Langfuse runs Python or TypeScript code evaluators on live observations, with a 2-second limit, no network access, and no third-party packages [9]. An evaluator can only check what the observation carries, so if a check needs the tool's raw output or the list of steps, record it on the observation the evaluator runs against. Checks that need the whole trace are easier in experiments, where your evaluator is ordinary code.

LLM judges: useful, biased, and in need of calibration

For questions code can't settle, like whether an answer stayed faithful to the retrieved documents, a second model grades the output against your rubric. Langfuse runs judges on observations, with filters on type, name, metadata, user, session, or tags, and a sampling rate to cap cost [10].

A judge is a model with failure modes of its own. The MT-Bench paper [11] documents position bias (favoring whichever answer comes first), verbosity bias (favoring longer answers), and self-enhancement bias (favoring output from its own model family). What helps in practice: a judge from a different model family than the agent, binary or small categorical labels instead of 1-to-10 scales, and a rubric that spells out concrete failure conditions.

Then calibrate it. Label a set of observations yourself and compare. Use Cohen's kappa rather than raw percent agreement, because raw agreement flatters a judge on imbalanced data. If 90% of your agent's answers are good, a judge that always says "good" agrees with you 90% of the time and has a kappa of exactly 0. Langfuse has a human vs AI annotation agreement view for this comparison. Skip calibration and you've traded tebak-tebak berhadiah for tebak-tebak berhadiah with charts.

Humans and users: the ground truth

Annotation queues give you or a domain expert a review inbox, and those labels become the reference set your judge is calibrated against. User feedback, thumbs up or down from the browser SDK or your server, is noisy and skews toward unhappy users. It's still the only signal that comes from the people you're building for, and a thumbs down is the best pointer to a trace worth reading.

Experiments and a CI gate that doesn't flake

Every low-scored trace is a real input your agent got wrong, which beats anything you'd invent at your desk as test material. Add it to a dataset, from the trace view or through the SDK, with an expected output where you have one. An experiment runs your current code over the dataset and scores each item with the same evaluators, so two prompt versions, two models, or two branches can be compared item by item.

Langfuse ships a GitHub Action, langfuse/experiment-action, that runs an experiment script on pull requests. The script raises RegressionError when a score falls below your threshold, the job fails, and the action can post the scores as a PR comment. It needs Python SDK 4.6.0 or JS SDK 5.3.0 and up [12].

from langfuse import Evaluation, RegressionError, RunnerContext

THRESHOLD = 0.9

def task(*, item, **kwargs):
    return support_agent(item.input["question"], user_id="ci", chat_id=item.id)

def correct_status(*, output, expected_output, **kwargs):
    ok = expected_output["status"] in output
    return Evaluation(name="correct_status", value=1.0 if ok else 0.0)

def experiment(context: RunnerContext):
    result = context.run_experiment(
        name="support-agent",
        task=task,
        evaluators=[correct_status],
    )
    scores = [
        e.value
        for r in result.item_results
        for e in r.evaluations
        if e.name == "correct_status"
    ]
    avg = sum(scores) / len(scores) if scores else 0.0
    if avg < THRESHOLD:
        raise RegressionError(
            result=result, metric="correct_status", value=avg, threshold=THRESHOLD
        )
    return result

The hard part is that threshold, because the score is a sample too. With 50 items and a true pass rate of 90%, the standard error is √(0.9 × 0.1 / 50), about 4.2 points, so the 95% interval is roughly ±8 points. An unchanged agent can score 86% on a bad day, and a fixed 0.9 threshold on a 50-item set will fail random pull requests until the team learns to rerun the job and ignore it. To keep the gate honest:

• Size the dataset for the change you want to catch. At 200 items the same interval narrows to about ±4 points.
• Compare against the main branch, not a constant. Score main on the same dataset and fail when the drop is larger than the noise.
• Give critical cases their own gate. An average can hide a case you just broke, so the Langfuse docs recommend checking each case against a reviewed baseline. Since a single run can flip on nondeterminism alone, run those cases k times and require pass^k.

Running it yourself

Self-hosted Langfuse is two containers, web and worker, on top of Postgres, ClickHouse, Redis or Valkey, and S3-compatible storage, with a Docker Compose file for a local trial [13]. Self-hosted v4 is generally available, and v3 gets security patches through January 2027 [7]. Traces hold your prompts, your users' messages, and whatever your tools returned, so if a contract or a regulator says that data stays on your servers, this is the path.

ClickHouse is the component that needs real operations work: disk, retention, merges. During the v4 migration, Langfuse's own team found partitions with around 1,000 parts where 150 to 200 is typical, and nodes learning about new data 25 minutes after insertion [5]. Budget for that, or use the cloud. The free Hobby plan gives 50,000 units a month, and a unit is any data point you send (the trace, each step, each score), so a 10-step run with two scores costs about 13 units, a little under 4,000 runs a month [14].

When this is overkill

In an earlier note I called a lot of AI tooling cosplay, evaluation harnesses included, when they get bolted onto tasks that don't need them. I still think that, so here's where the line sits.

If you use AI to write code, say Claude Code building five CRUD screens, you don't need Langfuse. The output is code, and you verify it the way you verify any code: read the diff, run the tests. Once it's merged, the model is out of the picture.

Langfuse earns its place when the model runs in production and its answers are the product: a support agent, a search over your documents, a classifier that routes tickets, a harness that runs agents for your customers. There every request is a fresh draw from the distribution, and nobody can review them all by hand.

Match the setup to the stage. With ten beta users, tracing alone is enough, and you'll read every trace anyway. Add evaluators once traffic outgrows your reading time. Add datasets and the CI gate when prompts change weekly and a silent regression would cost you a customer.

The setup I'd start with

1. Install Python SDK 4.7.0 or JS/TS SDK 5.4.0 or newer. Trace every request, set user_id and session_id at the top with propagate_attributes, and call flush() in anything short-lived.
2. Write code evaluators for trajectory failures: step budget, required tool calls, schema-valid arguments, parseable output.
3. Hand-label 100 observations. Then add one LLM judge for the quality question code can't answer, and check its kappa against your labels before you trust it.
4. Send every thumbs down and every low score to a dataset.
5. Gate pull requests that touch prompts, tools, or model settings on an experiment, with a threshold set from measured noise and a separate pass^k gate for critical cases.
6. Alert on steps per run and cost per run. A loop that stops terminating shows up there before it shows up on the invoice.

The takeaway

An agent that passes the five questions you tried has a failure rate somewhere below 45%. That's the honest reading of the evidence. Langfuse leaves the model exactly as smart as it was. What it gives you is evidence: traces that explain each failure, and scores with known error bars that can block a regression before a customer finds it. With both in place, shipping a prompt change stops being a quiz show.