Updated October 12, 2026, for laravel/ai 1.2. This post ran on laravel/ai 0.7.2. On 1.0 and later its generation middleware throws a
TypeErroron the first step, because agent middleware now wraps each step. It also read token fields that 1.0 renamed. The generation half now runs on two agent events, and the cost formula prices cached tokens separately. I re-ran the generation numbers on 1.2. I also fixed five errors that predate this update. See what changed.
Three weeks after you shipped the support bot, a customer asks about swapping a pair of shoes for a bigger size. The bot answers from your returns policy. Confident. Wrong.
You check the dashboards. Green. Latency normal, error rate flat, uptime at four nines.
You check the eval from part two. Still 94%.
But that 94% is from launch day. It scored 32 questions you wrote yourself, against a knowledge base nobody has touched since. It says nothing about the question that customer just typed.
Your retrieval is getting worse, and nothing in your stack is built to notice.
The series built a RAG pipeline on pgvector and the Laravel AI SDK, then measured it and reranked it. Every one of those checks ran offline, against a golden set, before you shipped. This post is the other half: watching retrieval on live traffic, where there are no labels and no test to fail.
Your stack watches everything except the thing that broke
Point Laravel Telescope at the request and you see the HTTP call to OpenAI. Status 200, a few hundred milliseconds, a blob of JSON. Pulse tells you the same call was a little slow. Your error tracker says nothing at all, because nothing threw.
A wrong answer and a right answer are the same shape. Both are a 200. Both cost tokens. Both come back fluent.
So the tooling you already trust is blind here by construction. It was built for a world where broken means an exception, and healthy means green. RAG breaks while staying green.
And the worst part is what your own code does with the evidence.
Here is the tool the support agent calls to search the knowledge base. This is the version from part one, running in production right now:
public function handle(Request $request): Stringable|string
{
$chunks = app(KnowledgeBaseRetriever::class)->retrieve($request['query']);
if ($chunks->isEmpty()) {
return 'No relevant policy information was found.';
}
return $chunks
->map(fn ($chunk) => "[{$chunk->source}]\n{$chunk->content}")
->implode("\n\n---\n\n");
}Look at what leaves the room. The retriever computed a cosine similarity for every chunk, 1 - (embedding <=> query), one number that says how close each match was. Then it mapped the chunks to strings and threw the number away.
The model never sees it. You never see it. The single most useful signal about whether retrieval worked gets computed on every request and dropped on the floor.
Keep the score
You do not need a new vendor to fix this. You need a table and a decorator.
The table is one row per query. What was asked, what came back, and the numbers the tool used to throw out:
Schema::create('rag_traces', function (Blueprint $table) {
$table->id();
$table->text('query');
$table->json('retrieved'); // per-chunk source, content, similarity, rank
$table->double('top_similarity');
$table->double('mean_similarity');
$table->double('min_similarity');
$table->double('embed_ms');
$table->double('search_ms');
// filled in later, once the model answers:
$table->text('answer')->nullable();
$table->unsignedInteger('input_tokens')->nullable();
$table->unsignedInteger('output_tokens')->nullable();
$table->double('cost_usd')->nullable();
$table->unsignedTinyInteger('judge_score')->nullable();
$table->timestamps();
});The decorator wraps the retriever you already have. Same retrieve(string, ?int) signature, so it drops into the tool with a one-line change. On the way through it times the embed and the search on their own, keeps the similarity scores, and writes the row:
class TracedRetriever
{
public function __construct(
private Embedder $embedder,
private KnowledgeBaseRetriever $retriever,
private TraceContext $context,
) {}
public function retrieve(string $query, ?int $k = null): Collection
{
$k ??= config('knowledge.retrieval_k');
$embedStart = hrtime(true);
$vector = $this->embedder->toVector($this->embedder->one($query));
$embedMs = (hrtime(true) - $embedStart) / 1e6;
$searchStart = hrtime(true);
$chunks = $this->retriever->search($vector, $k);
$searchMs = (hrtime(true) - $searchStart) / 1e6;
$this->record($query, $chunks, $embedMs, $searchMs);
return $chunks;
}
}Swap app(KnowledgeBaseRetriever::class) for app(TracedRetriever::class) in the tool and every live request now leaves a trace. Here is a real one, the shoes question, with each chunk’s text left out:
{
"query": "These shoes are too small, can I get the next size up without paying again?",
"top_similarity": 0.4144,
"retrieved": [
{ "source": "exchanges.md", "similarity": 0.4144, "rank": 0 },
{ "source": "returns-policy.md", "similarity": 0.3662, "rank": 1 },
{ "source": "damaged-or-defective.md", "similarity": 0.3327, "rank": 2 }
],
"embed_ms": 519.82,
"search_ms": 1.11
}Two things jump out of that row before you do any analysis.
The embed took 520 milliseconds. The pgvector search took one. The API call to turn the query into a vector is 99.8% of your retrieval time, and the nearest-neighbour search you were worried about is noise. A single “retrieval latency” number would have hidden that. Split the stages and it tells you exactly where to cache.
The second thing is the scores. And that is where the real signal lives.
A signal that needs no labels
Here is the honest part most observability posts skip.
Absolute similarity does not tell a right answer from a wrong one. On this knowledge base, the questions the bot gets right have a mean top-1 similarity of 0.52. The ones it gets wrong sit at 0.49. That gap is noise. Every support document is broadly about support, so a wrong-but-adjacent chunk scores nearly as high as the correct one. Do not build an alarm on it.
What the score does tell you, cleanly, is whether the corpus can answer the question at all.
I ran 42 real support questions and 10 questions the knowledge base has no business answering: weather, tax filing, the capital of New Zealand. The split is not subtle:
mean top-1 similarity lowest / highest
in-domain (42) 0.513 lowest 0.322
out-of-domain (10) 0.136 highest 0.251The lowest score any real question earned is 0.322. The highest any out-of-domain question reached is 0.251. They do not overlap. On this index, chunked at 600 characters, one threshold around 0.29 separates the 42 in-domain questions from the 10 out-of-domain ones.
That matters because retrieval never comes back empty. Ask something the corpus cannot answer and the agent still gets its nearest chunks as context.
{
"query": "What's the capital of New Zealand?",
"top_similarity": 0.073,
"retrieved": [
{ "source": "shipping-policy.md", "similarity": 0.073, "rank": 0 }
]
}Retrieval returned your shipping policy for a geography question, at 0.07, and nothing errored. Policy chunks that have nothing to do with the question are now the agent’s only context for its answer. Without the score, that is invisible. With it, you have a floor. On this index, anything under 0.29 is a query the corpus never covered, flagged before the model gets a chance to invent something. Alert on the rate of those and you find the gaps in your knowledge base by reading what your customers actually ask.
Re-check the floor whenever you re-chunk. The bigger-chunk run below dropped one real question to 0.286.
No golden labels. No test. Just a number you were already computing.
Watch it drift
The other thing a stored score buys you is time travel. You can compare today to last week.
Say a teammate wants to cut the embedding bill. Fewer, bigger chunks means fewer embedding calls, so they re-ingest with a larger chunk size and ship it. The build passes. Nothing errors. The knowledge base still has all nine documents.
Run a probe over your question set before and after:
php artisan kb:probe --set=support-questions.json
# mean top-1 similarity 0.543 (chunk size 600)
# ... re-ingest with --chunk-size=2000 ...
php artisan kb:probe --set=support-questions.json
# mean top-1 similarity 0.464 (chunk size 2000)Retrieval confidence dropped 15% from one config change. Every chunk now carries more text, so the vector is an average over more content, so it sits further from any specific question. The answers get vaguer. No exception fires. No offline eval catches it unless you happened to write a golden question for the exact topic that regressed.
The kb:probe command is a short loop: run a set of questions through the traced retriever, average the top-1 similarity, print it. Run it after any change that touches the index and you see the regression the same day, not three weeks later in a support ticket.
Tokens and cost on the same row
Retrieval is half the request. The other half is the model, and the Laravel AI SDK fires an event at the start and end of it.
Agent middleware is the wrong hook here. Since laravel/ai 1.0 it wraps each generation step, not the run. A support answer that searches first takes two steps, so the middleware runs twice and gets one step’s response each time.
The events see the whole run. PromptingAgent fires once before it. AgentPrompted fires once after it, with the final answer, the model id, and token usage summed across every step:
use Laravel\Ai\Events\AgentPrompted;
use Laravel\Ai\Events\PromptingAgent;
class TraceGeneration
{
public function __construct(private TraceContext $context) {}
public function start(PromptingAgent $event): void
{
if ($event->prompt->agent instanceof SupportAgent) {
$this->context->clear();
}
}
public function finish(AgentPrompted $event): void
{
if (! $event->prompt->agent instanceof SupportAgent) {
return;
}
$trace = RagTrace::find($this->context->id());
if ($trace === null) {
return; // the agent answered without searching
}
$usage = $event->response->usage;
$trace->update([
'answer' => $event->response->text,
'input_tokens' => $usage->inputTokens,
'output_tokens' => $usage->outputTokens,
'cost_usd' => TokenCost::usd($event->response->meta->model, $usage),
]);
}
}Register both methods in a service provider’s boot():
use Illuminate\Support\Facades\Event;
use Laravel\Ai\Events\AgentPrompted;
use Laravel\Ai\Events\AgentStreamed;
use Laravel\Ai\Events\PromptingAgent;
use Laravel\Ai\Events\StreamingAgent;
Event::listen([PromptingAgent::class, StreamingAgent::class], [TraceGeneration::class, 'start']);
Event::listen([AgentPrompted::class, AgentStreamed::class], [TraceGeneration::class, 'finish']);The streaming pair is not optional. ->stream() fires StreamingAgent and AgentStreamed, which extend the other two, but Laravel matches a listener to the event’s exact class, not its parent. Leave them out and a streamed answer never reaches its row. Nothing throws.
The tool writes the retrieval trace inside its own call. The listener runs once before the agent starts and once after it finishes. They never call each other, so a request-scoped TraceContext singleton carries the row id between them. start() clears it, the tool sets it when it writes the retrieval row, and finish() reads it when the answer comes back. That is how retrieval and generation land on the same row.
Now the two instanceof checks. Every agent fires these events, including the judge you’ll add in the next section. The judge runs after the response, in the same request, while TraceContext still holds the row id.
Drop the check and the judge’s own run writes its verdict over the answer it just scored, tokens and cost included. Then finish() samples that run as well, so the judge queues another judge. Take the check out of the companion’s test and it fails on that second judge call. The test runs the listener through the OpenAI gateway with HTTP faked, streamed run included.
There is no cost field in the response. There is no cost field in the OpenTelemetry conventions either. You compute it from tokens and your own rate card:
use Laravel\Ai\Responses\Data\TextUsage;
class TokenCost
{
public static function usd(string $model, TextUsage $usage): float
{
$rate = config('observability.pricing')[$model]
?? config('observability.pricing')['default'];
return round((
$usage->uncachedInputTokens() * $rate['input']
+ ($usage->cacheReadInputTokens ?? 0) * $rate['cache_read']
+ ($usage->cacheWriteInputTokens ?? 0) * $rate['cache_write']
+ $usage->outputTokens * $rate['output']
) / 1_000_000, 6);
}
}Don’t price inputTokens at one rate. On laravel/ai 1.x it includes cache reads and cache writes, and providers bill both at their own rates. One flat rate overstates every cache hit. uncachedInputTokens() is what’s left once you take both out.
Across the ten hard questions, each answer averaged 743 input tokens and 92 output tokens. The tokens are measured and real. The dollar figure is only as honest as the rate you put in the config, so put your model’s real numbers there. Now cost lives on the same row as similarity, and a query that quietly balloons to 4,000 input tokens because retrieval stuffed the context stops being a line item you find at month end.
Judge the answer you can’t see
Retrieval can be perfect and the answer can still drift from it. So score the answer too.
Offline, you judge against a policy document you wrote in advance. Production has no such document. What it has is the context the model was actually handed, which is enough for a reference-free faithfulness check: did the answer stay inside the chunks it was given?
Run it in a queued job, on a sample, off the request path:
class EvaluateRagTrace implements ShouldQueue
{
use Queueable;
public function __construct(public int $traceId) {}
public function handle(): void
{
$trace = RagTrace::find($this->traceId);
$context = collect($trace->retrieved)
->map(fn ($chunk) => "[{$chunk['source']}]\n{$chunk['content']}")
->implode("\n\n---\n\n");
$verdict = (new ResponseJudge)->prompt(<<<PROMPT
Question: {$trace->query}
Retrieved context (the only source of truth the agent was given):
{$context}
Support agent reply:
{$trace->answer}
PROMPT);
$trace->update(['judge_score' => $verdict['score']]);
}
}Sample it at the end of finish() so the customer never waits on the judge:
if (mt_rand() / mt_getrandmax() <= config('observability.judge_sample_rate')) {
EvaluateRagTrace::dispatch($trace->id)->afterResponse();
}Across the ten answers, the judge gave nine a clean 10 out of 10 for faithfulness and the tenth a 7, still a pass. That is a green result, and green is the point. You are not looking for a number that says the bot is broken today. You are building the one signal that tells you the day those scores start to drop, on traffic you never wrote a test for.
One request, one row
Four moving parts, one row. The retriever writes the retrieval half and stamps its id into TraceContext. The listener reads that id when the answer comes back and fills the generation half onto the same row. A sampled job adds the score last.
flowchart TD
Q["Customer question"] --> A["SupportAgent<br/>(traced by TraceGeneration on agent events)"]
A -->|calls tool| T["SearchKnowledgeBase"]
T --> R["TracedRetriever"]
R --> E["embed query"]
R --> S["pgvector search"]
E -->|"embed_ms"| ROW[("rag_traces row")]
S -->|"similarity, ranks, search_ms"| ROW
A -->|"answer, tokens, cost_usd"| ROW
ROW -->|"sampled, 1 in 10"| J["EvaluateRagTrace queued job<br/>judge_score"]
Read it top to bottom and it is one customer question turning into one durable record of what your RAG system actually did.
What you can now see
You started blind. A wrong answer looked exactly like a right one, and the only alarm was a customer complaint.
Now every request leaves a row. You watch top-1 similarity and catch the day retrieval drifts. You floor it at 0.29 and catch the questions your corpus can’t answer before the model bluffs. You read cost per query off the same line, and you score a sample of answers for faithfulness on live traffic instead of hoping.
When the table outgrows you, the escape hatches are there. A proxy like Cloudflare AI Gateway or Helicone gives you logs and cost for a one-line base-url change. The axyr/laravel-langfuse package auto-instruments the AI SDK for a real trace UI. Its v0.4.0 still targets laravel/ai 0.x and reads token fields that 1.0 renamed. Any OpenTelemetry backend takes the same spans through keepsuit/laravel-opentelemetry. Reach for them when a Postgres table stops being enough, not before.
The whole thing is on GitHub at MujahidAbbas/laravel-rag-evals, on the ai-1x branch for laravel/ai 1.2. The trace table, the decorator, the listener, and the probe command are all in it. Clone it, point it at your own knowledge base, and watch the number your retrieval has been hiding.
What changed
October 12, 2026, for laravel/ai 1.2:
- The post ran on laravel/ai 0.7.2. Its
TraceGenerationwas agent middleware that type-hintedAgentPrompt. Since 1.0, agent middleware wraps each generation step and receives aPendingStep, soTraceGenerationthrew aTypeErroron the first step. It also readpromptTokensandcompletionTokens, which 1.0 renamed toinputTokensandoutputTokens. The generation half now runs onPromptingAgentandAgentPrompted, which fire once per run. - The listener now ignores every agent except
SupportAgent. Without that check, the judge’s own run overwrites the answer it scores and queues another judge. The registration covers the two streaming events as well. - 1.0 counts cache reads and writes inside
inputTokens.TokenCostpriced all input at one rate. It now prices uncached input, cache reads and cache writes each at its own rate, and the rate card has two new keys. - I re-ran the generation eval on 1.2. The SDK’s OpenAI default moved from gpt-5.4 to gpt-6.1-sol, so that is the model it ran on. The averages went from 740 to 743 input tokens and from 100 to 92 output tokens, with no cache reads. The judge gave nine answers a 10 and one a 7. On 0.7.2 it gave all ten a 10.
- The retrieval numbers come from the embeddings API alone, which gets the same request on 1.2, so I didn’t re-run them.
- The repo link now points at the
ai-1xbranch. - Five older errors, unrelated to 1.x:
- The migration had no
output_tokenscolumn, though the 0.7.2 middleware wrote that field and the listener still does. It has one now, as the companion’s migration always did. - The
retrievedcolumn has to keep each chunk’s text, because the judge rebuilds its context from it. The post listed only source, similarity and rank. - The opening said part two’s eval scored 97% on ten questions. It scored 94% on 32.
- The out-of-domain section said the agent answers anyway and the request came back 200 OK. That eval ran only the retriever, so the section now says what it measured. It also scopes the 0.29 floor to this index, because the bigger-chunk run put a real question at 0.286.
- The closing said
axyr/laravel-langfuseinstruments the AI SDK. Its v0.4.0, from October 1, still targets laravel/ai 0.x.
- The migration had no