Your LLM can already do what Jev does. Should it?

Jev, from TypeSafe AI, takes some context and a question, and returns a probability for each answer instead of text. Which raises the obvious question:

“Can’t LLMs already do this with structured output?”

That is a fair question, because most of us already use LLMs for their contextual understanding and reasoning. So the real question is “where does an LLM still beat Jev, and what does that cost you?”

I tried it with an open model, reading the logits instead of decoding.

What does Jev actually do?

Jev is TypeSafe AI’s first product: a System One model that returns probabilities instead of tokens. A System One model makes “fast and intuitive” decisions, as opposed to System Two models, which make “slow and deliberate” ones.

The Jev model takes in a “state” and a “question” as input. A “state” provides the model with context that the “question” should address. A “question” includes the question itself, a set of options, and a type of decision that the model should make. Noul, choice, and score are the three types of question. Noul is a yes/no decision and returns a probability of yes, choice returns a distribution over the options, and score returns a probability-weighted value on a 2–10 scale. One thing worth knowing before you try it: Jev is hosted only. A request can hold 64k tokens in total, enough to ask several questions about the same state at once, but the state plus the longest question must fit in 32k of them. It costs $0.042 per million input tokens.

The timing of its introduction is interesting. Not long ago, most companies used pre-trained models like BERT for these decisions, because of their contextual understanding. Then LLMs came along, and companies moved to them for two reasons: they were simply better than BERT, and using one was an API call instead of deploying a model yourself. That made the switch a no-brainer, and structured output, or JSON with a single field, covered the rest. The catch is that LLMs are still large, and not optimised for System One tasks. Talk about hiring a chef to make toast.

That is where Jev comes in: it keeps the simple API call, but is built for the System One tasks that LLMs are too big for. The gains you’d get depend on how much overhead you are carrying by using LLMs on System One tasks like classification.

Why logits instead of structured output?

The first thing I, along with many other engineers, thought of was to skip structured output and just use the logits. Structured output still decodes the answer token by token. Reading the logits skips decoding entirely, so you run one forward pass and read the probabilities straight off the final layer. With an LLM, we just read the logits over the option tokens and apply a softmax to get a probability distribution over the options. mini-jev, an open project that tests exactly this, measured it against structured output on Qwen3-4B: the same accuracy, about 4× faster on short texts.¹ It’s simply a matter of loading the model via Hugging Face or vLLM and invoking the forward pass to get the logits. And if you want the same API as Jev, you can just add a post-processing step:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen3.5-4B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to(device)

def decide(state: str, question: str, options: list[str]) -> dict:
    letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    option_lines = "\n".join(f"{letters[i]}. {o}" for i, o in enumerate(options))
    prompt = (
        f"State:\n{state}\n\n"
        f"Question:\n{question}\n\n"
        f"Options:\n{option_lines}\n\n"
        "Answer with the letter only."
    )
    # Chat template with thinking off, so the next token is the answer letter.
    text = tokenizer.apply_chat_template(
        [{"role": "user", "content": prompt}],
        tokenize=False, add_generation_prompt=True, enable_thinking=False,
    )
    inputs = tokenizer(text, return_tensors="pt").to(model.device)

    # One forward pass, no decoding: read the logits at the last position.
    with torch.no_grad():
        logits = model(**inputs).logits[0, -1]

    # Logits for the letter tokens only, then softmax.
    letter_ids = [tokenizer.encode(letters[i], add_special_tokens=False)[0] for i in range(len(options))]
    probs = torch.softmax(logits[letter_ids].float(), dim=-1)

    winner = int(probs.argmax())
    return {
        "choice": options[winner],
        "probabilities": {
            options[i]: round(probs[i].item(), 4) for i in range(len(options))
        },
    }

How did it compare?

On quality, it depends on how big your state is. With states up to 32k tokens, Jev comes out ahead of a plain logit readout: on typed-decisions, a public benchmark of exactly these kinds of decisions, Jev scores 0.727 against 0.626 for Qwen3-8B.

Part of that gap is that an LLM doesn’t treat every option fairly out of the box. Evening that out is a big enough topic for its own post: Every decision model plays favourites, Jev included. AnyJev, a tool that turns any LLM into a System One model like Jev, does that work in levels of optimisation, and the results get better at each one. On the same benchmark, Qwen3-8B goes from 0.626 to 0.647 at L0 (evens out the options, no labels), 0.648 at L1 (tunes the confidence scores), and 0.771 at L2 (incorporates a small classifier), past Jev.² L1 barely moves the accuracy; what it improves is how far you can trust the confidence scores.

Past 32k tokens, there is no contest. I tested my own code wrapping Qwen3.5-4B on a private routing dataset, where the system prompt alone takes the prompt to 64k tokens. With some light tweaking to even out the options, it reached 0.786. Jev simply rejected those requests. That is the other advantage: you’re able to fit in a system prompt and even a larger “state” and “question” set, as Qwen3.5-4B takes 262k tokens natively,³ against Jev’s 32k for the state plus the longest question.

	State and question up to 32k tokens	Over 32k tokens
Jev 1.13	0.727	rejected
LLM, reading logits	0.626	0.786
AnyJev L0	0.647
AnyJev L1	0.648
AnyJev L2	0.771

Latency depends on your setup. For me, with Qwen3.5-4B on an L4 GPU, a single forward pass on those 64k-token prompts could take up to a few seconds, even without any decoding. Of course this depends on the model size, type, and GPU that you use. For a sense of Jev’s side, my own requests through OpenRouter took 0.30 seconds at the median, but those states were only about 660 tokens long, so the two numbers aren’t directly comparable.

Which one does your workload need?

Use your LLM if your state and question go over 32k tokens, or if you need the highest quality, which takes AnyJev’s higher levels. The caveats are latency, and that you have to host the model yourself, at least for now. For everything else, Jev is the better option.

In production, at scale, it comes down to what you’re optimising for:

• Over 32k tokens? Your LLM. Jev rejects the request.
• Need to keep your data in-house? Jev is hosted only. If you need something like Jev, you can consider Laya, von, or NanoJev. Or go ahead with the LLM approach.
• Latency-sensitive? Jev. TypeSafe claims 70–500 ms end to end,⁵ the typed-decisions maintainers measured 710 ms at the median,⁴ and my own requests took 0.30 seconds. Laya is faster still, at about 33–40 ms on a T4 GPU by its authors’ numbers, but only takes prompts of up to 1,024 tokens.⁶
• Price-sensitive? Jev, at $0.042 per million input tokens, with output free.⁷ An L4 GPU at about $0.89 an hour⁸ only costs less if you push more than about 21 million input tokens an hour through it.
• High volume? Jev’s limits are 1,200 requests a minute and 250,000 tokens a second.⁷ Plan around those before you commit.
• Data-rich? If you have plenty of past decisions to learn from, AnyJev’s higher levels on your own LLM gave the highest quality above, and confidence scores you can trust. How that works is in the follow-up post.

The other alternatives to Jev

Jev and your own LLM aren’t the only options. A few open alternatives have appeared since Jev launched:

• Laya: a ModernBERT model tuned and packaged as an alternative to Jev. It needs fine-tuning to be strong (0.362 zero-shot on typed-decisions, 0.766 fine-tuned) and takes prompts of 512 to 1,024 tokens.⁶
• von: another open Jev alternative built on ModernBERT.⁹
• NanoJev: scores each option on its own path, so the order of the options can’t sway the answer.¹⁰
• mini-jev: the same logit readout as the code above, with a local server.¹
• AnyJev: covered above, and in the follow-up post.

Keep in mind that the open alternatives only appeared in the past few weeks.

On the closed side, meraGPT’s Decider 1 scores 0.768 on typed-decisions, but that result was submitted by meraGPT itself.⁴

Thoughts

The moat for LLMs is still their contextual knowledge. They simply contain far more trained information, with so many more parameters. We don’t know how big Jev is, what base model it’s built on, or how it’s trained, so we don’t know how well it generalises to nuanced cases, or whether it handles them better than an LLM.

Also, if you already have a model deployed on an H100, I do think there’s a case for the LLM variant of Jev over Jev itself, once vLLM exposes the same model through a /v1/systemone endpoint so the GPU is shared. vLLM already merged a prototype example server for that endpoint, for DiffusionGemma only, and it is not a standard vLLM endpoint yet.¹¹

Footnotes

1. mini-jev, 2026-09. ↩ ↩²

2. AnyJev results, Nokia Applied Research, 2026-09-22. ↩ ↩²

3. Qwen3.5-4B model card, Qwen, accessed 2026-09-29. ↩

4. LocalLLaMA/typed-decisions, leaderboard as of 2026-09-22. ↩ ↩² ↩³

5. Introducing System One models and Jev, TypeSafe AI, 2026-09-15. ↩

6. Laya model card, Convai Innovations, accessed 2026-09-28. ↩ ↩²

7. Models, TypeSafe docs, accessed 2026-09-28. ↩ ↩²

8. NVIDIA L4 pricing, getdeploying, median on-demand price across providers as of 2026-09-23. ↩

9. von, 2026-09-24. ↩

10. NanoJev, 2026-09-17. ↩

11. vLLM PR #57250: structured generation mode for DiffusionGemma (Jev-like), vLLM, merged 2026-09-22. ↩