Jev and the agentic web

the experiment

How we ran it

The site is a warehouse, a helpdesk and a document vault we host and seed ourselves, so every task has a known answer or end state. The agents are simple Claude agents, with Claude Haiku 4.5 writing any free text. The only difference between the arms is who makes each step's closed choices: Jev, or Claude Haiku 4.5 from the same state. Ten tasks, five repeats each, a fresh workspace per run: 240 runs, each verified against live state or the exact answer.

01 / time

Faster in every paired run

Browser tasks took 1.9 s with Jev and 8.2 s without. WebMCP: 1.4 s against 3.7. NLWeb: 2.0 s against 7.8. The slowest run with Jev still beat the median run without it, on every surface.

02 / cost

Up to 13× cheaper per task

Without Jev, deciding is nearly the whole cost: $0.0220 of a $0.0233 browser task. With Jev it costs $0.0007, and writing the text becomes the largest cost.

03 / task success

Success held, except on three browser tasks

WebMCP and NLWeb passed every task, with Jev and without. The browser passed 78% of runs with Jev and 71% without, and the whole gap comes from three tasks.

04 / the browser

Jev won the longest browser task and lost the simplest

Jev finished the seven-step ticket task in every run and gave up on the two-step lookup in every run, where the language model found the search box. Its wins and its failures repeat exactly: with Jev, every task took the same path on all five repeats.

05 / where the time goes

Deciding was most of a task. With Jev it is a small part.

Without Jev, choosing the next step is 66 to 83% of a task's time and nearly all of its cost. Jev answers that choice in about 150 ms for a fraction of a cent, so deciding drops to 13 to 37% of the time, and what remains is the one or two calls that write the text. The writer is the same, and on WebMCP and NLWeb so is the number of steps. On the browser Jev also took fewer steps, 4.8 a task against 6.2, because it never waited or repeated a click. What is left is the writer and the page itself.

06 / conclusions

What we conclude

Jev cuts a step decision to about 150 ms, so the agent's own thinking is no longer where a task's time or cost goes. What Jev cannot do is finish a task the page does not put in front of it: it will not guess a search box it was never offered, or a second step that is not reachable from the first. Both of its failures in this study were one of those, and both tasks passed in every run where the site offered the same action as a WebMCP tool.

1. 01
   With Jev deciding, agents finished tasks up to 4.3× faster, and were faster in all 120 paired runs.

   1.9 s against 8.2 on the browser, 1.4 against 3.7 on WebMCP, 2.0 against 7.8 on NLWeb. The slowest run with Jev beat the median run without it on every surface.
2. 02
   A task cost up to 13× less.

   $0.0018 against $0.0233 on the browser, $0.0011 against $0.0046 on WebMCP, $0.0029 against $0.0175 on NLWeb. Without Jev, deciding alone was $0.0220 of a $0.0233 browser task.
3. 03
   Success held. On the browser the arms split on three tasks, two of them in Jev's favour.

   WebMCP and NLWeb passed every task both ways. On the browser Jev won the seven-step ticket task 5 to 0 and the price lookup 5 to 3, and lost the reporter lookup 0 to 4, because the search box the language model used was not among the options Jev was offered. Jev never waited and took the same path on every repeat.
4. 04
   Both of Jev's failures were browser tasks the page did not put in front of it.

   The reporter lookup needed a ticket that was not in the visible list and a search box Jev was never offered. The transfer needed a second step that was not reachable from the first. Exposed as WebMCP tools, both tasks passed in every run with either decider. With deciding this cheap, the largest gains are in what the site exposes to agents.
5. 05
   What the site exposes decides whether a task finishes. The decider decides how fast and at what cost.

   WebMCP had the smallest gain from Jev, 2.6×, and was the only surface where every task passed both ways. The browser had the largest gain, 4.3×, and every failure in the study. A tool removes a decision before the decider sees it, so the surface with the most exposure gains least from a faster decider and still finishes first: 1.4 s a task on WebMCP with Jev, against 1.9 s on the browser.
6. 06
   Jev fails fast and the same way every time. The language model fails slowly and differently each time.

   Jev's two browser failures took the same path on all five repeats; on the reporter lookup that was one click, then it gave up, in 2.4 s. The language model failed the ticket task by clicking Change status four times in a row, 8 steps and 10.7 s, and passed the price lookup 3 of 5 with runs from 9.7 to 20.7 s. A failure that repeats exactly is one a site owner can find and fix once. What remains after a cheap decider is the text the model writes, 62 to 90% of a Jev task's cost, so the next lever is fewer free-text steps: tool arguments over typed text.

07 / sources