Project Name: ZelusBench Your Team: Individual — Arsh Chawla
"Can the model focus on what matters and ignore what doesn't?" The competition poses this for attention, but answering it requires a prior question many existing benchmarks fail to confront: what does it even mean for something to matter?
In natural language, relevance is negotiable. Whether a clause is a distractor, whether a detail is "critical context," whether a rule should override a habit, these judgements bleed into interpretation, world knowledge, and memorized priors. This makes attention nearly impossible to isolate. A model that looks distracted may simply disagree about what's important; a model that looks focused may just be pattern-matching familiar phrasings.
ZelusBench grounds relevance in geometry. In a 3D coordinate space, whether a point's definition matters to a distance query is not interpretation, it is a mathematical fact, algorithmically verifiable. By building attention tasks on deterministic spatial relationships, we construct a controlled environment where attention failures become measurable rather than inferred.
The payoff is conceptual as much as methodological. Consider how we typically probe shifting attention in language, redefining a word or character mid-passage: "from now on, call Alice 'Bob'." The trouble is that model performance there confounds attention with vocabulary priors, entity frequency, and training distribution. A model that "fails" to track the rename may simply have a stronger prior on "Alice" than the local instruction can overwrite. Geometry strips this away. Reflecting a space over the y-axis is a pure mental-set shift: no token has become more or less familiar, no semantic network has been reshuffled. What remains is the attention mechanic itself, clean of vocabulary bias, unbounded by the size of the model's memorized lexicon, and uncontaminated by how often the test phrasing appeared in pretraining.
The same argument extends to selective attention (geometric noise points are unambiguously relevant or not, unlike "distractor sentences" whose distractor-ness is up for debate) and sustained attention (a chain of spatial relations has a single correct terminal state, unlike narrative tracking where drift is partly stylistic).
ZelusBench procedurally generates a 3D space and poses queries with deterministic answers. By varying the parameters of point generation, we construct tasks that map directly onto the three canonical attention faculties.
Core task. Points are introduced through chains of relative relationships, "G is 2.8 units from F in the direction (0.6, 0.4, 0.8)", forcing the model to build a representation incrementally rather than read off absolute coordinates. It is then asked a deterministic question: What are D's coordinates? Is A closer to B or C? Which point is farthest from origin? Every answer is single-valued and geometrically derivable.
Parameterized difficulty isolates each faculty:
- Selective attention: we inject noise points defined in chains irrelevant to the final query. The noise ratio (low / medium / high) scales filtering demand.
- Sustained attention: we increase chain depth, lengthening the relational path between the origin and query-relevant points. This probes spatial-state maintenance without drift.
- Shifting attention: mid-scenario, we apply a geometric transformation (reflection, rotation, translation) that invalidates the prior coordinate map. The model must discard its representation and rebuild. Complexity scales reconstruction cost.
The codebase is modular: space generation, parameter application, prompt assembly, and verification are cleanly separated. Exact rational arithmetic is used throughout to avoid floating-point contamination of ground truth.
The dataset is 100% synthetically generated, eliminating contamination risk. Generation is fully seed-deterministic, ScenarioConfig + seed = identical scenario. Nothing is pre-stored; every scenario is regenerated on the fly at evaluation time.
Scale. Nine shipped tasks: three attention axes (selective / sustained / shifting) × three tiers (short / medium / long). Each task sweeps 2 pinned levels of its target knob × 10 seeds = 20 scenarios per task, 180 total (~540 POSITION queries).
Pinned knobs per task. Everything else is drawn from ScenarioConfig.randomize_except() with a deterministic background RNG, so the same seed index produces the same background across tiers, any score delta isolates the pinned axis, not background variance.
| Axis | Pinned | Short | Medium | Long |
|---|---|---|---|---|
| Selective | num_points = round(5 × k) at depth 5 | k ∈ [1.0, 1.5] | [2.0, 3.0] | [4.0, 5.0] |
| Sustained | min = max_chain_depth = d | d ∈ {3, 6} | {9, 12} | {15, 18} |
| Shifting | transform_prob = p at depth 6 | p ∈ [0.0, 0.1] | [0.2, 0.3] | [0.4, 0.5] |
All shipped tasks use dim=3 and POSITION queries targeting deep points (query_min_depth ≈ depth − 2, or placed after a transform for the shifting axis). Each scenario exposes its prompt, the ordered point/transform/query trace, and a dependency DAG that resolves to closed-form ground-truth coordinates. Verification reads off the DAG, no LLM judge.
Engine. Space holds a DAG of PointDefinition objects; resolving a point is a topological walk with numpy.float64 arithmetic. Seven point types (cartesian_offset, magnitude_direction, magnitude_polar [2D], magnitude_spherical [3D], midpoint, weighted_centroid, projection) and four transforms (rotation, translation, reflection, scaling), each propagating through all dependents. 79 unit tests lock engine correctness, if the engine is right, ground truth is right by construction.
Prompt format. Each statement is a single natural-language line in chronological order. Transforms are plain prose ("Rotate Point C, Point E by 120° around (0,0,0) on axis (1,0,0)"), no delimiter, no banner. The model must notice the redefinition from content, not formatting. Answers are requested as [Answer q_ID] value (coordinate tuple, scalar, or label).
Parsing. [Answer q_ID] tags are extracted first, falling back to [Query q_ID] splits, then full text. Within each block, coordinate / distance parsers use the last regex match, so reasoning scratch-work like "midpoint ≈ (1,2,3) so…" is skipped and only the final answer is scored. Unparseable output is scored REFUSED = 0.0 and tracked separately from wrong answers.
Scoring. Deterministic, tiered, grader-free:
| Query | Metric | EXACT (1.0) | CLOSE (0.7) | APPROX (0.3) |
|---|---|---|---|---|
| POSITION | absolute Euclidean error | < 0.5 units | < 2.0 | < 5.0 |
| DISTANCE | relative error | < 1% | < 5% | < 15% |
| BOOLEAN | exact match | binary | - | - |
Absolute (not relative) thresholds on POSITION are deliberate: a 5-unit drift should cost the same near origin as far from it, so chain length can't trade against accuracy.
We evaluated several frontier models on ZelusBench, revealing clear gradients in performance and unique failure modes. The overall scores (see table in appendix) ranged from ~0.79 (Gemma 4 31B) down to ~0.30 (GPT-5.4 mini), demonstrating the benchmark's discriminatory power. More importantly, the sub-task scores paint a detailed cognitive profile for each model.
Key Insights:
-
Selective Attention is a Major Weak Spot: Most models showed significant degradation as noise increased. For example, Claude Opus's score dropped from 0.91 (selective_low) to 0.53 (selective_high). This quantifies a real "distractibility" under informational clutter, a failure mode predicted by attention theory (Chapter 3: Attention).
-
Sustained Attention Reveals Context Limits: Performance on sustained_long tasks varied dramatically. Gemini 2.5 Flash collapsed to 0.33, while Claude Sonnet and Opus maintained ~0.83–0.94. GPT-5.4 models showed a severe downward trend (0.90 → 0.12 for GPT-5.4), graphically illustrating a context degradation or "forgetting" of early relations within a long chain, a phenomenon directly analogous to issues measured in long-context research (Context Degradation in LLMs).
-
Shifting Attention is Challenging Across the Board: No model excelled at high-complexity shifting. Even the best (Gemma 4 31B) scored only 0.5 on shifting_high. This task, requiring the model to abandon a carefully constructed mental map and build a new one, proved exceptionally difficult, highlighting a rigidity in reasoning that is not exposed by static question-answering.
-
Model-Specific Profiles Emerge:
- DeepSeek V3.2 showed an intriguing anomaly: exceptional shifting_low (0.97) but poor selective_low (0.52). This suggests its architecture or training may prioritize tracking changes over filtering static noise.
- Gemini 2.5 Flash exhibited a stark contrast between strong selective filtering with low noise and a catastrophic failure with high noise (0.90 → 0.21), indicating a brittle filtering mechanism.
- The GPT-5.4 family showed uniformly low scores, with a particularly steep decline in sustained attention, suggesting a potentially different architectural prioritization or a weakness in the chain-of-thought reasoning required by our task.
These results confirm that ZelusBench provides a meaningful signal. It doesn't just rank models; it explains why by attributing performance to specific, isolated cognitive faculties.
ZelusBench offers a novel contribution by operationalizing the abstract concept of "relevance" through geometry. This moves attention evaluation from qualitative observation to quantitative measurement. Its novelty lies in:
- Deterministic Grounding: Unlike benchmarks with ambiguous or cultural answers, relevance is mathematically defined. A point either matters to the query or it does not.
- Decomposition of Attention: It directly maps to the tripartite cognitive model of attention (selective, sustained, shifting), allowing for targeted diagnostics rather than a single composite score (Four Types of Attention).
- Contamination-Proof Design: Fully synthetic generation guarantees the tasks are novel, ensuring we measure attention, not recall.
The discriminatory power is evident in the results. The benchmark produces a wide spread of overall scores and, more importantly, reveals divergent patterns of strengths and weaknesses across sub-tasks. A model can score similarly overall to another but have a completely different cognitive profile (e.g., strong sustainer vs. strong shifter). This granularity allows researchers to ask not just "which model is better?" but "which model is better for what kind of thinking?", a crucial step toward building detailed cognitive profiles for AGI development.
Developed independently.
- Plomecka et al. Measuring Progress Toward AGI, Cognitive Abilities (2026).
- Chapter 3: Attention; Four Types of Attention — selective/sustained/shifting taxonomy and cognitive control.
- Context Degradation in LLMs — long-context recall failure.
- Related spatial-reasoning work: SnorkelSpatial, GeoGramBench, SpatiaLQA.
- Deterministic vs Stochastic ML.
Methodological Foundations:
- Deterministic vs Stochastic ML Fundamentals — on deterministic system design.
- Stochastic and Deterministic Clarifications — clarifying key concepts for evaluation.