Every decision model plays favourites, Jev included

What is a decision model?

A decision model takes a state, the context the decision is about, and a question with a set of options, and it returns a probability for each option instead of text. Jev, TypeSafe AI’s hosted System One model, is one; the last post covers what it does.

There are three ways to get one:

• Call a hosted one: Jev, or meraGPT’s Decider 1.¹
• Run a small model trained for it: Laya² and von,³ both built on ModernBERT, or NanoJev, with an architecture of its own.⁴
• Wrap an LLM you already run: instead of decoding an answer, read the probability of each option straight off the model’s logits. AnyJev, a Python package from Nokia Applied Research, does this for any LLM. mini-jev does it on Qwen3-4B,⁵ and vLLM has a prototype /v1/systemone server.⁶

Why does an LLM play favourites?

In the last post, we addressed various reasons for having LLMs be decision models. But to fully realise their potential, we have to address the elephant in the room: bias.

By simply reordering the options (e.g. order C, B, A instead of A, B, C) passed to an LLM based decision model, 23% of answers are shown to have swapped.⁷ How often the answer changes when only the order of the options does is called the Flip Rate.

The favouritism comes from two biases. Both are in the nature of LLMs: token bias, where some option tokens are more likely to begin with, and ordering bias, where the position of an option in the list changes its score. Using an LLM as a classification engine means correcting them first, then calibrating the confidence.

Ordering bias

LLMs are causal by nature, so earlier tokens get seen by everything after them, and attention favours the start of the sequence.⁸ If the model uses rotary position embeddings (RoPE), the recent tokens would get more attention instead.⁹

So are RoPE-based LLMs favouring earlier or later tokens? That depends. Both of these characteristics act together. This is consistent with the paper “Lost in the Middle”, which found that how well LLMs use information forms a U-shape across positions: best at the start and end of the context, worst in the middle.¹⁰

Additionally, the balance between favouring earlier or later tokens is model-dependent, as shown in “Beyond Early-Token Bias”: Qwen2.5-7B-Instruct, DeepSeek 7B Chat and Mistral 7B all favour late positions.¹¹

Token bias

Turning an LLM into a Decision Model requires using unique tokens to represent each option. However, token selection in LLMs is inherently biased to begin with.

The model favours some label tokens before it has read a single option. Research on LLMs handling multiple-choice questions has shown this.¹² If we drop using label tokens altogether, the bias drops. If we keep the label tokens and hold the ordering of the options, but randomly assign the label tokens to each option, the bias only dips a bit.

gpt-3.5-turbo, 0-shot	Bias left	RStd, MMLU	RStd, ARC
Letters A/B/C/D in order	token and ordering	5.5	3.3
No letters, the model picks the option text	ordering only	1.0	0.6
Letters randomly assigned, options in place (e.g. B/D/A/C)	token only	5.1	3.7

The favoured letter is not always A: on balanced MMLU, llama-30B picks A/B/C/D 34.6/27.3/22.3/15.8% of the time, while gpt-3.5-turbo picks them 22.5/25.6/32.3/19.6%, favouring C.¹²

Share of answers	A	B	C	D
llama-30B	34.6%	27.3%	22.3%	15.8%
gpt-3.5-turbo	22.5%	25.6%	32.3%	19.6%

Fine-tuning on multiple choice data whose answer letters are unbalanced introduces the bias too.¹³

The others play favourites too. Sort of.

Being causal isn’t the whole story. You’d think a BERT-like model with bidirectional attention, where every token sees every other token, would have no ordering bias. But the same issue shows up in Laya, which is a ModernBERT model tuned and packaged as an alternative to Jev.¹⁴ Von, another open Jev alternative built on ModernBERT, flips too, at least until version 1.2.³ The options still sit in one sequence: with RoPE, an option further down the list sits further from the question, and each option is read alongside the others, so its neighbours change its score.

So what does escape it? Only a change of architecture, and both examples are research releases. von 1.2 stops the options from seeing each other and restarts their positions, and NanoJev scores each option on its own path.³⁴ Training for calibrated confidence isn’t enough: Laya is trained with RLCD, the same calibration method Jev claims,¹⁵ and it still flips.

What about Jev?

Since its architecture is undisclosed by TypeSafe and they did not publish any order-invariance claim, I did a test to find out. I sent the same state with the option list reversed and counted how often the answer changes. Jev’s answer flips 11% of the time. The states and questions come from typed-decisions, a public benchmark of business decisions from customer service, invoice processing, security incidents and agent traces.¹

JEV 1.13 · SAME STATE, SAME QUESTION · ONLY THE OPTION ORDER CHANGES

How should this vendor invoice be dispositioned?

• Invoice $149,888 for 400 units, matching the purchase order
• Delivery accepted with exceptions, 0 units received
• Payment due in 2 days

SlotOriginal orderReversed order

1approve0.06reject0.00

2holdcorrect answer0.61manual_review0.62

3manual_review0.33holdcorrect answer0.34

4reject0.00approve0.04

Answer: holdAnswer: manual_review

Bars: the probability Jev gives each option.

Isn’t that just noise? Sending the same request again without reordering flips only 2% of answers.

Second request, same state and question	Answers changed	Flip rate
Options reversed	33 of 300	0.11
Options in the same order, sent again	7 of 300	0.02

Model	Flip rate	Measured how
von 1.1	0.495	own hard tier, options reshuffled (any of 4 orderings)
von 1.2	0.0	retrained with options blinded, so they can’t see each other
Laya	0.04–0.23	by task, permuted options
Jev 1.13	0.11 (rerun: 0.02–0.03)	reversed options, 100 typed-decisions states (my test)

So Jev isn’t immune either: whichever model you use, the options need evening out.

How do you even out the biases?

AnyJev has implemented some methods to handle these biases already. Their implementation handles them with rotations and priors as their L0 optimisation. They have two more levels of optimisation on top of L0: L1 and L2.⁷

Level	Needs	What it does
L0	nothing	rotates the options, and divides out the model’s average bias (the prior)
L1	100–500 labelled examples per question	L0, along with a selected temperature that tunes the confidence scores
L2	20–300 labelled examples per question	a small classifier reads the model’s hidden state, in place of the logits

L0: rotations and a prior

With L0, we send the question once per option, rotating the list by one each time, so every option sits in every slot, under every letter, once.¹² The prompts share the same state, so they run together in one batch, and each option’s probabilities are averaged across the rotations into one answer. That evens out the ordering bias, and any bias towards a letter.

However, the rotation only evens out the slots, and the letters tied to them. What about the token bias in the answers themselves? That’s what the prior is for: the model’s average answer to the question, over every request it has seen, used from the 8th request on. Each new answer is divided by it, model ÷ prior^0.75, so an answer that only matches the model’s average bias loses its edge.

Why 0.75 and not a full division? The prior only sees the model’s answers, never the right ones. Say the model answers “approve” on 74% of invoices. It could be biased towards “approve”, or 74% of invoices could really deserve approval. Without labels, AnyJev can’t tell the two apart. A full division treats all of it as bias, so when one answer really is that common, it pushes the model off answers it was right about. 0.75 stops short of that: across 230 model and question pairs, it gave the best average gain and halved the worst loss.¹⁷

L0 cuts the bias rather than removing it: on BANKING77, the flip rate drops from 23% to 7%.⁷ The cost is one prompt per option, so a 20-option decision takes about 16.6 times as long as a single pass.¹⁸ An opt-in rotation budget cuts that by stopping once the answer is settled, after about 7 of 18 rotations on average.¹⁹

L1: a temperature for the confidence

L0 evens out the answers, but a model that’s overconfident is still overconfident after it. L1 fixes that with one number per question, a temperature T: each probability is raised to the power of 1/T and renormalised. A T above 1 softens the confidence, below 1 sharpens it, and the top answer never changes.

There is one T per question, shared by all of its options, and no model to train. AnyJev runs L0 on 100 to 500 labelled examples, tries values of T, and keeps the one that gives the correct options the most probability, scored by log-likelihood, which punishes a confidently wrong answer far more than an unsure one.¹⁶

On BANKING77, accuracy barely moves, but the expected calibration error (ECE), how far the confidence is from how often the model is right, halves from 0.184 to 0.095.⁷ Why that matters is the next section.

L2: a classifier in place of the logits

The logits are the model’s guess at the next token across its whole vocabulary, and its last blocks are busy turning the answer into that token.⁷ L2 skips them. It stops the forward pass about two thirds of the way in and hands the hidden state, the model’s working summary of the prompt, to a small linear classifier that AnyJev calls a head. The head is trained on your labelled examples for that question and outputs one probability per option, so there is no letter to read off and no rotation to run.

It’s more accurate and cheaper. On typed-decisions, Qwen3-8B goes from 0.647 at L0 to 0.771 at L2, past Jev’s 0.727.⁷ It needs one prompt per decision, not one per option, and the forward pass stops early. It’s the old BERT trick of training a scikit-learn classifier on the [CLS] vector,²⁰ applied to a decoder.

The catch is the labels. A head is tied to one question and its set of options, so a new option set needs new labelled examples. It matches L0 at 20 labelled examples and passes Jev at about 100.²¹ It also reads the hidden state, so it only works on a model you run yourself, and it still flips 7% of the time.

How a question moves up the levels

We don’t have to decide when a question moves up. Every question starts at L0, and its prior keeps updating with every request. As labelled examples come in, from human review, outcomes, or the LLM being replaced, AnyJev solves the head at 30 and moves the question to L2, then re-solves it at 60, 120, and so on.²¹

After that, the head looks after itself. If the question is reworded, the head keeps its weights and only re-learns the average and spread of its inputs from the next 30 requests, with no labels. A new option set is a new question, and starts again from labelled examples. L1 calibration is manually triggered and usually done in batches, whereas the L2 label observation can be done each time a decision call is made.

ONE DECIDER · LEVEL="AUTO" · THREE QUESTIONS

“Which team?” never got labels, so it stays at L0, which still evens out the options with nothing.

Which team?L0

Prior

divided out, from 42 requests

Labels

none

Temperature

none

decide()

How urgent?L1 + L0

Prior

frozen: measured on the 200-example batch

Labels

200, in one batch

Temperature

T = 1.4, from calibrate()

decide()

Refund?L2

Prior

not used: the head reads the hidden state

Labels

155: head re-solved at 60, 120

Temperature

bypassed: T = 1.3 from calibrate()

decide() · observe() ×5

Why calibration matters more than accuracy

A model you can trust is a model you can automate. Calibration of the decision model equates to more accurate confidence scores, which equates to better trust in the model itself. The more confident we are of how the decision model behaves, and how predictable it is, the easier it is for engineers to design a system around it. Being able to trust the results also means engineers can act on them more confidently.

Two metrics check how far the confidence scores can be trusted, each answering a different question:

• Can you take a score at face value? The expected calibration error (ECE) compares how confident the model says it is with how often it’s right. If the answers given at 80% confidence are right 80% of the time, there’s no gap. ECE averages that gap over all the answers, so 0 is perfect.
• Does the score tell you which answers to trust? Sort the decisions from most to least confident, and let the model take them in that order until 5% of the ones it has taken are wrong. The share it takes is automatable; the rest go to a person. It can’t go much higher than the accuracy, so a big gap between the two means the scores, not the answers, are holding the model back.

Take two models that each get 18 of 20 decisions right. Model A is wrong only where it was least sure; Model B is wrong where it was most sure:

20 decisions, 18 right	Where the 2 mistakes sit	Accuracy	Automatable at ≤5% error
Model A	its 2 least confident answers	90%	90%
Model B	its 2nd and 10th most confident answers	90%	5%

BANKING77	Accuracy	ECE	Automatable at ≤5% error
Qwen3-8B, raw logits	0.747	0.240	7.7%
+ AnyJev L0	0.803	0.184	46.3%
+ AnyJev L1	0.807	0.095	52.0%
Jev 1.13	0.867–0.870	0.066–0.070	72.7–76.7%

typed-decisions	ECE
+ AnyJev L0	0.15–0.40
+ AnyJev L1	0.036–0.055
+ AnyJev L2	0.03–0.05
Jev 1.13	0.144

From raw logits to L1, accuracy moves 6 points, but the automatable share goes from 7.7% to 52.0%. In a queue of 10,000 support tickets, that’s 770 routed without a person against 5,200. Jev, the best calibrated here, routes 7,300 to 7,700.

While accuracy helps to prove that the model works, calibration and reliability are what make the model usable in production and yield business benefits.

Conclusion

An LLM changes 23% of its answers when only the order of the options changes, the ModernBERT alternatives flip too, and so does Jev, 11% of the time. Biases in decision models shouldn’t be ignored. There’s more to be done to reduce the bias and improve the accuracy, but beyond that, calibrating the confidence scores is something we shouldn’t leave out.

There are also newer systems I’ll be excited to read up on and test out:

• Clef, from Cloudflare: two open-weight decision models, 27B and 9B, with the same API as Jev, hosted on Workers AI.²²
• Nimble, from Bespoke Labs: a 9B decision model fine-tuned from Qwen3.5-9B, which runs locally through Ollama’s new decision endpoint.²³
• Decision 2.0, from the vLLM Semantic Router project: a family of open-weight decision models from 0.6B to 27B.²⁴

Their launch pages report accuracy and speed, but none of them reports a calibration error or a flip rate yet.

Footnotes

1. LocalLLaMA/typed-decisions, leaderboard as of 2026-09-22. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

2. Laya model card, Convai Innovations, accessed 2026-09-28. ↩

3. von README and option_marker.py, 2026-09-24. ↩ ↩² ↩³ ↩⁴

4. NanoJev invariance check, 2026-09-17. ↩ ↩²

5. mini-jev, 2026-09. ↩

6. vLLM PR #57250: structured generation mode for DiffusionGemma (Jev-like), vLLM, merged 2026-09-22. ↩

7. AnyJev README and results, Nokia Applied Research, 2026-09-22 to 2026-09-28. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

8. On the Emergence of Position Bias in Transformers and Efficient Streaming Language Models with Attention Sinks, arXiv, 2025-02-04 and 2023-09-29. ↩

9. RoPE decay and attention sinks in position bias, arXiv, 2025-08-21. ↩

10. Lost in the Middle, Liu et al., arXiv, 2023-07-06. ↩

11. Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs, Menschikov et al., arXiv, 2025-05-22. ↩

12. Large Language Models Are Not Robust Multiple Choice Selectors, Zheng et al., arXiv, 2023-09-07. ↩ ↩² ↩³ ↩⁴ ↩⁵

13. Selection bias introduced by fine-tuning, arXiv, 2024-06. ↩

14. Laya BENCHMARKS.md, 2026-09-27. ↩ ↩²

15. Introducing System One models and Jev, TypeSafe AI, 2026-09-15. ↩

16. AnyJev v0.2.0 source (anyjev/decider.py, anyjev/calibrate/), Nokia Applied Research, 2026-09-28. ↩ ↩² ↩³ ↩⁴

17. When does L0 help?, AnyJev docs, Nokia Applied Research, v0.2.0, 2026-09-28. ↩

18. AnyJev latency results, Nokia Applied Research, 2026-09-21. ↩

19. AnyJev rotation budget, Nokia Applied Research, 2026-09-27. ↩

20. A Visual Guide to Using BERT for the First Time, Jay Alammar, 2019-11-26. ↩

21. AnyJev method v3, Nokia Applied Research, 2026-09-22. ↩ ↩² ↩³ ↩⁴ ↩⁵

22. Introducing Clef: Cloudflare’s first open-source decision models, now on Workers AI, Cloudflare, 2026-10-01, and Clef model card. ↩

23. Ollama now supports Jev-style decision models, Ollama, 2026-09-29, and nimble, Ollama library. ↩

24. Decision 2.0, vLLM Semantic Router, Hugging Face, and Decision-2.0-Vega-27B model card, accessed 2026-10-04. ↩