I tried Jev on a simple question: can it choose a bill based on someone's priorities? Mostly, yes. But rearranging the options changed more decisions than I expected.
It followed the instructions in 106 out of 120 scenarios. Reversing the option order changed 16 decisions. Asking the exact same question again changed only two. Its overall score barely moved, because it fixed some mistakes and made new ones.
This was a fun side experiment. I've been studying how institutions turn what people want into decisions. My first paper used simulations to examine how legislative rules affect which bills pass and how closely they match people's preferences. My second, Who May Overrule the Agent?, looked at when an AI reviewer should be allowed to replace another agent's decision.
So when Jev appeared, I had a small test I wanted to try: before getting into parliaments or AI reviewers, can it follow one person's instructions?
Why try Jev?
It's pretty hyped right now on X; everyone seems to be trying it. Apparently I also needed a Jev side quest.
Jev takes a question and a set of possible answers, then returns a choice and probabilities. Basically, an AI answering multiple-choice questions. TypeSafe's explanation.
Kitze reported adding it to Skillbox to find relevant tools, while Rasmic said he was using it to label training data. Good Start Labs tested it for grading another AI's work. These are early reports; I haven't independently tested their setups. Skillbox, data labeling, grading experiment.
Takuya made a fair point: older models could already do many of these tasks. The interesting question is what becomes practical when those decisions get faster and cheaper. His post.
I already had a small piece of my research I could try it on. Give it someone's priorities and see what it chooses.
A simple test
Suppose someone says:
Public transport matters most. Housing comes next. Energy policy comes third.
The rule is explicit: satisfy the first priority, then use the second and third to break ties.
If two bills both improve public transport, but only one matches the person's housing preference, that bill wins because of the order this person gave us.
The made-up people in the test had different priorities about housing, transport and energy. Each chose between three bills and leaving things unchanged. Every option had known consequences, so ordinary code could identify the correct answer. Jev received the instructions and options, never that answer.
Real preferences are messier. This simpler setup asks whether the AI follows instructions when the rules are clear.
After a small trial, I fixed the larger test before running it: 120 new scenarios, each asked in several ways: unchanged, repeated exactly, with reversed options, with renamed bills, and with a committee chair recommending another option. The chair explicitly had no authority to change the person's instructions.
I also included equivalent device-setting questions, without political language. Altogether, the trial and main test used 936 requests.
Similar scores, different choices
| What we changed | Correct answers | Decisions changed |
|---|---|---|
| Original question | 106/120 | — |
| Asked it again unchanged | 104/120 | 2 |
| Reversed option order | 104/120 | 16 |
| Renamed bills | 103/120 | 5 |
| Added chair recommendation | 104/120 | 6 |
Reversing the list fixed seven mistakes but spoiled nine correct answers. The score barely changed, yet 16 people received a different decision.
If an AI represents you, you'd probably want your answer to survive someone rearranging the list.
The order effect was larger than ordinary repeat variability. The smaller differences from renaming bills and adding a recommendation were less convincing. This isn't strong evidence that a chair can manipulate Jev.
The first trial made reversing the list look helpful. That didn't hold up in the larger test. I'm glad I checked before recommending it.
Were its probabilities useful?
Yes, in this sample.
When Jev assigned its choice at least 90% probability, 66 of 67 answers were correct. Below that threshold, 40 of 53 were correct.
Reviewing the lower-probability answers would therefore have flagged 13 of the 14 mistakes while checking fewer than half the answers. I calculated that from the saved responses; I haven't tested how well a person would handle the review.
One wrong answer still received 96% probability. A 96% prediction can fail, so that alone tells us little about whether the probabilities are sensible. We'd need to compare lots of predictions with their outcomes.
I think the scores helped identify harder cases here. I'd want more testing before trusting them on another task. These results use the probability assigned to the chosen option; Jev's separate confidence field measures how concentrated the probabilities are. TypeSafe's distinction.
What does this tell us?
I wouldn't read this as a finding about real voters or political bias. Jev also made mistakes on the device-setting questions, although it answered 111/120 correctly. That comparison changed several wording features, so it doesn't isolate the effect of politics.
It does connect to the question behind my other work: when someone acts on your behalf, what gets preserved and what changes? Here, something as small as the order of a list sometimes changed the choice. I haven't tested how that would affect a whole institution.
For this exact task, I'd use ordinary code. The rules are already explicit. A future role for AI might be extracting preferences from messy language, then letting code apply the rules—but we haven't tested that approach here.