Can a specialist model choose an agent's tools?

AI agents often choose from a growing list of tools. Could Jev, a model built for structured decisions, make that choice faster and cheaper?

Setup

Both tests used the same 100 synthetic prompts: 47 read tasks, 21 needing approval, 13 ambiguous requests, 10 needing no tool, and 9 the tools could not handle. The 16 inert mock tools covered GitHub, browser, local files, and calendar, four each. Results are medians, with ranges below. All six reports have complete cost data. Costs are estimates based on configured token prices, not bills.

The reports used jev-latest (responses identified jev-1.13.0) and openai/gpt-5.6-luna through OpenRouter. Requests were sequential, alternating provider or arm order by case. Jev had a three-second routing timeout; on fallback, Test 2 gave Luna all 16 tools. Luna had a 30-second timeout. Both SDKs disabled retries. The client machine and network location were not recorded. Cost estimates apply reported input and output tokens at $0.042 and $0 per million for Jev, and $0.20 and $1.20 for Luna. Test 2 includes routing and agent calls; the reports carry exact figures. Provider-native tool search was not tested, so the all-tools arm is a floor for comparison.

Test 1: The ideal world — Jev makes the routing decision

TypeSafe advertises Jev as 193.6× faster and 444.6× cheaper than LLMs on its workflow benchmark, while noting these figures are likely at the high end of real-world gains. An API can force a named tool with tool_choice, but the LLM still generates its arguments; the agent decides whether to call a tool and when to stop. I built Jev Tool Router to measure routing separately, then test calling Jev before the main LLM.

Jev and Luna each chose a tool for every prompt; no agent call followed. Tool fit means the selected tool suited the request. It was measured on 81 prompts with an acceptable tool and does not measure task success; some prompts still needed clarification.

Both routers received the same request state, options, yes/no questions, two-stage process, and decision thresholds. Luna received a system prompt and strict JSON output format requiring a choice, probabilities, confidence, and yes/no judgments; Jev used corresponding Choice and Noul questions. Neither request specified temperature or maximum tokens. The repo does not record comparative tuning effort.

Jev chose a suitable tool for a median 76 of 81 requests (range 76–76); Luna did so for 77 of 81 (76–77). Jev took 419 ms per request at the median (run averages 405–426 ms), versus 2,459 ms for Luna (2,383–2,539 ms). Estimated cost per 100 requests was about $0.0049 ($0.00484–$0.00486) versus about $0.037 ($0.03658–$0.03685).

Jev was faster and cheaper in all three routing runs. It tied Luna on tool fit once and trailed by one case twice. Jev produced fewer correct routes ready to execute: 41 (range 41–41) versus 46 (45–46). Tool fit and readiness are different judgments.

Test 2: A mock agent — Jev filters tools before Luna

This is what can be built today. One version offered Luna all 16 mock tools. The other called Jev before each turn and offered Luna its selected tool, no tool, or the full list on fallback. Luna still chose whether to call a tool and answered the prompt. Both versions used the same prompts and inert mock results; no external service ran.

Both arms used the same Luna model, instructions, context, tool schemas, mock results, and four-turn limit. Luna used automatic tool choice, with no explicit temperature, response format, or maximum token count.

On 47 read tasks, the Jev arm succeeded on a median 37 (range 36–37), versus 35 with all tools (35–38). The all-tools arm won one run, 38 to 37, so Jev did not improve success consistently. Across all 100 prompts, median successful outcomes were 62 (60–63) with Jev and 55 (54–56) with all tools.

Mean read-task latency was 3.737 seconds with Jev at the median (range 3.565–4.001), versus 3.011 seconds with all tools (2.842–3.204), an increase of about 24% from the all-tools median. Jev was slower in every run. These means include failed attempts. Across all 100 prompts, mean latency was 2.488 seconds (2.381–2.629) versus 2.340 (2.191–2.481).

Estimated cost per 100 requests fell to about $0.021 with Jev ($0.02081–$0.02114), from about $0.028 with all tools ($0.02841–$0.02934): about 26% less at the medians. Jev cost less in all three agent runs.

Jev asked for clarification on 12 of 13 ambiguous requests in every run, versus 6–7 with all tools. The shorter tool list itself may make Luna more likely to ask, so this may not be a router effect. Both arms struggled with approval cases (2–4 of 21 successes with Jev; 2–3 with all tools) and unsupported requests (0–1 of 9; 0 of 9). These scores use simple checks, not a full review of answer quality.

What I take from this

To me, these results suggest a place for specialist models alongside general LLMs in agents. In the routing-only test, Jev took about 83% less time and cost an estimated 87% less than Luna at the medians. That does not show how Jev would perform as a model's native tool router. In the mock agent test, Jev lowered estimated API cost by about 26% but increased read-task latency by about 24%, with no consistent gain in completed tasks. I would be cool to test a setup where the routing step itself - the decision being made to call a specific tool - can be configured, whether through a provider API or an open-source agent stack.

This was a small test with synthetic prompts, 16 inert tools, simple answer checks, and three runs. Test 1 routed only the initial prompt; real agents choose tools again as results arrive. Test 2 used a bounded mock loop, not real services. More tools might strengthen the case for prefiltering, while the extra routing call takes a larger share of latency with only 16. I did not test provider-native tool search, a stronger baseline than the all-tools arm. These results do not establish statistical significance or predict production performance.

The experiment guide has the methods and reports. Earlier 1.5-second routing and agent reports are historical; those agent runs lacked complete Jev cost data. Older 86-request routing and ten-task agent tests used different requests. I have not mixed them into these results.