Intuition
Can agents anticipate the outcome of an experiment before running it?
Leaderboard
GPT-6 Astra
—
—
Claude Opus 5.5
—
—Claude Fable 5.1
—
—
Gemini 3.8 Flash
—
—
Kimi K3
—
—
Grok 4.7
—
—
DeepSeek V4.1 Flash
—
—Pareto frontier
By task
GPT-6 Astra
—
Claude Opus 5.5
—
Claude Fable 5.1
—
Gemini 3.8 Flash
—
Kimi K3
—
Grok 4.7
—
DeepSeek V4.1 Flash
—
FrontierRSI is a benchmark that asks an agent to build a useful learned capability under a fixed research budget. It designs and runs experiments, interprets the results and submits a complete system. Independent tests score what that system can do.
The ten tasks
Pokémon Battle Post-Training
- Builds:
- A trained model that chooses attacks and switches.
- Final test:
- Complete battles against a diverse opponent league.
- Metric:
- Match results, opponent-balanced, higher is better.
Baba: Learning From Experience
- Builds:
- A player that learns from attempts and keeps useful knowledge.
- Final test:
- Unfamiliar puzzles after an autonomous learning stream.
- Metric:
- Puzzles solved, plus learning gain, higher is better.
Nano Jev
- Builds:
- A small, general-purpose decision engine.
- Final test:
- Protected tool, routing, relevance, evidence and reasoning cases.
- Metric:
- Decision score, 0–100, higher is better.
Wristband Handwriting Decoding
- Builds:
- A model that reads handwriting from wrist muscle signals.
- Final test:
- Transcription for people absent from training.
- Metric:
- Character error rate, lower is better.
Two-Picture Holograms
- Builds:
- A generator that turns two images into a phase mask.
- Final test:
- Both reconstructed pictures in an independent optical simulator.
- Metric:
- Reconstruction quality, dB, higher is better.
MarketLab
- Builds:
- A trained trading system that can update causally.
- Final test:
- After-cost performance through a protected market period.
- Metric:
- Risk-adjusted, after-cost return, higher is better.
Selective Representation Control
- Builds:
- A learned controller for text and visual representations.
- Final test:
- Intended changes, downstream consequences and preserved facts.
- Metric:
- Edit success with preservation, higher is better.
Circuit Repair Post-Training
- Builds:
- A model that investigates faults and chooses repairs.
- Final test:
- Restored circuit function under unfamiliar conditions.
- Metric:
- Repairs restoring all functions, %, higher is better.
Virtual Knockout
- Builds:
- A model of gene-perturbation responses.
- Final test:
- Withheld genes and gene–cell combinations.
- Metric:
- Response error vs. no-change, lower is better.
Experimental NMR Prediction
- Builds:
- A model of experimental chemical shifts.
- Final test:
- Assigned measurements for unfamiliar molecules.
- Metric:
- Shift error vs. experiment, lower is better.
Towards formalising research taste
We are working to make research taste measurable. Alongside the raw task scores, we measure subtler capabilities, orthogonal to those scores, that shape an agent’s taste. These four are the first; many more are coming.
Judgement
Can they scale an experiment down, and extrapolate the result back up?
Efficiency
How good a result do they reach for the compute they spend?
Control
How well do they use compute: estimating costs, provisioning and staying within limits?
Working with us
Beyond measuring these capabilities, we are working on how to train for them, starting with machine learning and moving across the empirical sciences. If you would like your models to be better at research, write to us at founders@syntheticsciences.ai.