Evaluating and Benchmarking the System One Model Jev

Jev (jev-1.13.0) is a commercial "System One" model from Type Safe AI that does not generate text: given a state and typed questions, it returns a choice from fixed options, a position on a rubric, or the probability that a statement is true, with probabilities described as calibrated.

Known limitations

• All three models degrade on low-resource languages, fine-grained or noisy labels, and rubric-based judgments.
• Binary probabilities need tuned thresholds; untuned micro-F1 on UNFAIR-ToS was 0.50.
• Probes rule out shallow memorization but not memorized question-answer pairs.
• Jev completes 35% of ALFWorld's unseen tasks, 42% of 90 Text World games, and 21% of BabyAI-Text missions; Qwen completes more ALFWorld and Text World games (54.5% and 51.1%) but shares its failures on BabyAI-Text and Frozen Lake.

Examples

• Small decisions in information access pipelines, such as classification, routing, moderation, and rubric scoring.

Research · arXiv cs.AI · Publisher website: arxiv.org · Last verified 5 November 2026 · Catalog ID 130040