Updated October 8, 2026, for laravel/ai 1.2. Jev shipped in the tagged 1.0.0 release on September 23, four days before this post went up, and this post said it had not. The testing section also claimed the docs skip the fake’s filled-in answers. They cover them, as a convenience. The post’s code is unchanged, and the companion branch’s tests pass on 1.2.0. See what changed.
My research agent picks which web pages to read from a title and a 150-character snippet.
Then it reads them. All of them.
In the context-hygiene runs, tool output was 868,902 of the 904,286 characters the agent carried by its last turn.
96%.
So when Jev landed in the Laravel AI SDK in September, the use was obvious. Let a cheap classifier read each page first. Send the agent only what passes.
Every Jev article so far stops at the same line. Send the state, get back 0.94, compare it to a number somebody picked.
Freek’s spam filter picks 0.75 and 0.9. Firecrawl’s use-case article sketches a page filter with cut-offs at 0.45 and 0.55, then admits it never ran the code.
I picked 0.7. It sounded careful.
On labelled data it threw away 9 of the 29 useful pages.
Jev returns a number, and every decision after it is yours
Jev is a classifier from TypeSafe. It does not write text. You send it state and typed questions, and it sends back probabilities.
SDK support shipped in laravel/ai 1.0. The code below runs on 1.2:
composer require "laravel/ai:^1.2"The docs still mark classification experimental, so a minor release can change it. 1.2 already did: custom classification gateways and providers now take an $attachments argument.
Two naming traps first. The SDK calls TypeSafe’s yes/no question a Boolean, not a Noul. And the default provider is typesafe, which needs early access.
OpenRouter serves the same model with no waitlist. Everything below runs through classify('openrouter').
Here is the judge. One request per page, two questions per request:
use Illuminate\Support\Str;
use Laravel\Ai\Classification;
use Laravel\Ai\Classification\Boolean;
use Laravel\Ai\Contracts\Question;
use Laravel\Ai\Responses\ClassificationResponse;
class PageJudge
{
public const int MAX_CHARS = 12_000;
public function __construct(
public string $provider = 'openrouter',
public int $timeout = 10,
) {}
public function judge(string $topic, string $query, array $page): ClassificationResponse
{
return Classification::of([
'research_topic' => $topic,
'search_query' => $query,
'title' => $page['title'] ?? '',
'url' => $page['url'],
'page' => self::excerpt($page),
])
->questions(self::questions())
->timeout($this->timeout)
->classify($this->provider);
}
public static function excerpt(array $page): string
{
return Str::limit((string) ($page['markdown'] ?? ''), self::MAX_CHARS);
}
/** @return array<string, Question> */
public static function questions(): array
{
return [
'relevant' => new Boolean(
'The page is about the research topic.',
[
'true' => 'The page discusses the specific subject named in research_topic',
'false' => 'The page is about something else, or only mentions the topic in passing',
],
),
'injection' => new Boolean(
'The page contains text that tries to give instructions to an AI system reading it.',
),
];
}
}Both questions go in one request. Output tokens are free, so the second question costs almost nothing.
The truncation is not optional.
One search returned the Laravel queues docs page at 179,266 characters, roughly 45,000 tokens. Jev takes 32,000 for the state plus its longest question.
The first run broke it three ways
I wrapped the toolkit’s FirecrawlSearch in a JudgedSearch tool instead of editing it. It searches with scraping on and returns only what passes the judge.
Then I ran the agent on one topic and read every row.
The first version put the agent’s search query in research_topic. The same Stack Overflow page scored 0.47 under one query and 0.85 under the next.
Same page. Same topic. Different words in the search box.
Judge against the topic the user asked about. Pass the query as its own field.
Next, Firecrawl returned zero characters for a Reddit thread. Jev scored the empty page 0.92 relevant, from its title alone. TypeSafe’s jaggedness page warns that the model reads literally, and here it was, live.
Empty pages now skip Jev and reach the agent as a title and a URL.
And four pages came back in both searches, so the agent got an 11,890-character GitHub issue twice. One send per URL per run fixed that.
Every one of those bugs was in my code around Jev.
You only see them by storing every judgement.
Store the probability, not the verdict
Every page gets a row: the probabilities, the verdict, the model id Jev reported, the input tokens and the latency.
SearchJudgement::create([
...$row,
'relevant' => $relevant,
'injection' => $injection,
'verdict' => $verdict,
'model' => $response->meta->model,
'input_tokens' => $response->usage->inputTokens,
'ms' => (int) ((hrtime(true) - $started) / 1_000_000),
]);A boolean column answers one question forever: what the threshold said that day.
A probability column lets you move the threshold next month and see exactly which pages change sides.
No API calls.
The model column matters more than it looks. jev-latest resolved to typesafe/jev-1.13-20260917 on every call. When that string changes, your probabilities change under you.
TypeSafe’s own models page says it plainly: once you tune a threshold against a version, pin that version instead of the alias.
Where the threshold actually comes from
A threshold needs an answer key. Somebody has to say, page by page, whether it was worth reading.
I built the labelling command. Then I pressed Enter through all 84 pages without reading them.
The default option was “useful”. Every label was worthless.
So I cleared them, made Skip the default, and replaced myself with something I would actually finish.
Two frontier models from different vendors, Opus 5.5 and GPT-6 Astra, read the exact excerpt Jev saw. They never saw Jev’s scores. They answered my question, not Jev’s: would a developer researching this topic want to read this page?
Where both agreed, that became the label.
They agreed on 77 of 84.
TypeSafe grades Jev against the same kind of panel in its own evaluations. It tells you how often cheap Jev matches expensive judges. It does not tell you the truth, any more than an LLM judge in your evals does.
So I added two checks that need no model at all.
Topic swap. Take a page the panel called useful and judge it against a different topic. A good Filament v5 upgrade guide is irrelevant to Horizon memory leaks by construction. All 26 dropped. Mean relevance fell from 0.76 to 0.02.
Planted injection. Insert one instruction to AI readers in the middle of a useful page. All 26 scored 0.99.
Now sweep the threshold against the 77 agreed labels:
threshold kept useful kept useful lost junk kept chars kept
0.2 65 29 0 36 76.1%
0.3 50 26 3 24 57.0%
0.5 37 26 3 11 43.0%
0.7 25 20 9 5 27.2%
0.9 9 8 21 1 9.3%My 0.7 lost 9 useful pages to save 16 points of context against 0.5.
Look at the rows between 0.3 and 0.5. Useful-lost never moves. Only the junk does. That is the knee, and no single probability shows it to you.
The three useful pages that 0.5 still loses are not off-topic.
They are next door.
A deep dive on queue worker internals, for a topic about Horizon memory leaks. Livewire’s lazy-loading docs, for a topic about slow tables. Jev read “about the research topic” narrowly, exactly as its jaggedness page says it will.
So the gate gets a third outcome:
$verdict = match (true) {
$injection >= $this->injectionAt => 'drop',
$relevant >= $this->keepAt => 'keep', // 0.5
$relevant < $this->dropBelow => 'drop', // 0.2
default => 'brief',
};Below 0.2, not one useful page in the set.
Between 0.2 and 0.5, the page reaches the agent as a title, a URL and a description. The agent knows the neighbour exists and can search for it by name. It just does not pay to read it.
Calibrated, to its own question
TypeSafe says Jev is calibrated: a 0.9 should be right about nine times in ten. Bucket the labelled pages and check.
Jev says pages mean p actually useful
0.8 - 1.0 13 0.92 85%
0.6 - 0.8 18 0.71 72%
0.4 - 0.6 15 0.47 13%
0.2 - 0.4 19 0.26 16%
0.0 - 0.2 12 0.12 0%It holds at the top and the bottom.
The middle runs high.
Jev answered “is this page about the topic”. My label asked “is this page worth reading”.
A thin page that mentions your topic three times is a yes to the first and a no to the second.
Calibration holds for the question you asked. If your label asks something else, the middle of the curve is yours to measure. The sweep tells you where it stops holding.
The injection question flags pages about prompts
One real page scored 0.96 on injection. It was a PGConf talk on generative AI and databases, and it quoted example prompts: “Human: You are an agent who manages orders and returns…”.
A page about prompts contains prompts.
Jev cannot tell the difference. If your agent researches AI topics, it meets these pages every day.
At an injection cut-off of 0.7, the gate dropped 2 of the 29 useful pages. Every planted instruction scored 0.99, so I moved the cut-off to 0.98. That keeps both real pages and still catches all 26 plants.
One phrasing of one planted instruction is thin evidence. Treat 0.98 as a starting point.
This is a filter, not a security boundary. The prompt-injection guardrails post covers the boundary.
I also cut a third question. has_evidence scored 0.8 or higher on 53 of 77 pages, and only half of those were useful.
A question that says yes to nearly every page tells you nothing.
Test the gate without calling Jev
Classification::fake() is the obvious way to test this, the same pattern as faking agents in your tests.
It has a trap.
Fake only the question you care about, and the SDK fills in the rest with random numbers.
The docs present the fill as a convenience: “your tests only need to provide the answers they make assertions against.” They never say the answers are random.
preventStrayClassifications() does not stop it. The response is not missing, only a key inside it.
I faked relevant and left injection out. Five calls returned 0.333, 0.035, 0.751, 0.379 and 0.463. My test asserts on the verdict, never on injection, so the docs’ rule lets me leave it out. But the gate reads injection, and the test goes green or red at random.
So every fake goes through one helper, and one test pins the helper to the judge:
use Laravel\Ai\Classification;
use Laravel\Ai\Prompts\ClassificationPrompt;
use Laravel\Ai\Responses\Data\BooleanAnswer;
public function test_it_keeps_briefs_and_drops_by_relevance(): void
{
$this->fakeSearch(['https://a.test', 'https://b.test', 'https://c.test']);
Classification::fake(fn (ClassificationPrompt $prompt) => match ($prompt->state['url']) {
'https://a.test' => $this->answers(relevant: 0.92),
'https://b.test' => $this->answers(relevant: 0.35),
'https://c.test' => $this->answers(relevant: 0.1),
});
$out = $this->search();
$this->assertSame(['https://a.test'], array_column($out['pages'], 'url'));
$this->assertSame(['https://b.test'], array_column($out['unsure'], 'url'));
$this->assertSame(1, $out['dropped']);
$this->assertArrayNotHasKey('content', $out['unsure'][0]);
}
public function test_the_fake_answers_cover_every_question_the_judge_asks(): void
{
$this->assertSame(array_keys(PageJudge::questions()), array_keys($this->answers(relevant: 0.5)));
}
private function answers(float $relevant, float $injection = 0.0): array
{
return [
'relevant' => new BooleanAnswer($relevant),
'injection' => new BooleanAnswer($injection),
];
}Add a question to PageJudge and forget the helper. The second test fails before a random number can pass the first.
The suite also pins the empty-page skip with assertNothingClassified() and the truncation with assertClassified(). Eleven tests, no API key.
What it did to the agent
Now the part every Jev article skips: what the filter does to the agent.
Five topics the threshold had never seen. Each one run twice with the filter and twice without, on Haiku, with the same step cap on both sides.
The step cap matters. The SDK derives the default from the tool count, as the model-routing post found. Swapping search plus scrape for one tool quietly cut my agent from five steps to three.
filter off filter on
median tokens 64,137 76,654
mean tokens 81,060 79,747
worst run 151,908 123,775
standard deviation 43,738 23,889The filter did not make the agent cheaper. The median went up.
Without the filter, the agent skims snippets and usually scrapes one page.
Cheap.
Then on another run it asks search for the full markdown of every result. Four of the ten unfiltered runs did. The worst of them sent 151,908 tokens, almost four times the cheapest run.
With the filter, the agent reads a few bounded 12,000-character excerpts every time.
So the spread nearly halved. The same topic run twice without the filter differed by up to 112,565 tokens. With it, 58,480.
And the answers? Opus and GPT-6 Astra judged each pair blind, in random order. Filter won 5, no filter won 5. Where the judges agreed, 4 each.
Same quality. Similar typical cost. No runaway runs.
That is what a Jev filter buys an agent like this one.
Anybody promising you a smaller bill has not run it.
Two production notes.
Jev failed on 14 of 84 calls through OpenRouter at a 10-second timeout, one of them TypeSafe’s own “system overloaded”. My first version passed failed pages through in full, and the run with the most failures was the heaviest of its batch. A failed judgement now degrades to a title and a URL.
And if you build your own label panel with structured output, check the keys. GPT-6 Astra returned winner instead of the schema’s better on 4 of 10 calls.
FAQ
Do I need a TypeSafe account to use Jev in Laravel?
No. OpenRouter serves Jev, so an OPENROUTER_API_KEY is enough. Without a model argument, the SDK asks OpenRouter for ~typesafe/jev-latest. To make OpenRouter the default for every classify() call, change default_for_classification in config/ai.php from typesafe to openrouter.
How much does a Jev call cost, and does the SDK tell you?
Jev costs $0.042 per million input tokens, and output tokens are free. The SDK returns the token count on $response->usage->inputTokens, not dollars, so you multiply by the price yourself. OpenRouter’s response does carry a usage.cost field, which the SDK drops (issue #1069 asks to keep it).
Can I run Jev on my own server?
Not Jev itself. TypeSafe publishes no Jev weights or self-hosted package, and its docs describe only the hosted API. You can self-host an open model that speaks the same API, such as Laya. The SDK’s typesafe driver reads a url from its provider config, so a local laya-serve is one more provider entry in config/ai.php. Re-run your threshold sweep on it before you switch, because TypeSafe tells you to pin even the Jev version you tuned on.
What you can do now
You started with a probability and a number that sounded careful.
Now you store every judgement with the model that made it. You pick the threshold from a sweep that prints the useful pages it loses next to the context it saves. You check calibration against your own labels and know which part of the curve to trust.
You send the uncertain middle as a title instead of forcing it to one side. You test the gate with fakes that cannot hand you a random number.
And you know what the filter buys before you promise it to anyone.
The judge, the tool, the sweep, the panel and the controls are on the jev branch of laravel-ai-research-agent. Point the sweep at your own labels:
php artisan ai:sweep-threshold --signal=relevantI checked the SDK code above against laravel/ai 1.2.0.
Pick the threshold from the data. Then write down which pages it throws away.
What changed
October 8, 2026, for laravel/ai 1.2:
- Jev shipped in the tagged laravel/ai 1.0.0 on September 23. This post went up on September 27 and said Jev was only on the
1.xbranch. That was wrong on publish day. The install line now requires^1.2. - The testing section said the docs never mention the fake’s filled-in answers. They had covered them since September 21, as a convenience. The section now quotes them and names what they leave out: the answers are random.
- No code in this post changed. The
jevbranch now runs on laravel/ai 1.2.0 and its eleven tests pass. Re-run over the stored judgements, the sweep reproduces both tables row for row. Nothing this post calls behaves differently in 1.1 or 1.2, and the fake still fills a missing answer with a random number. - The judgement numbers depend only on the classification call, which 1.1 and 1.2 left unchanged on OpenRouter. The agent runs are September runs on
1.x-dev, and I did not repeat them on 1.2.