The one-paragraph version
People increasingly ask language models for chances — how likely is a market move, a disaster, a life event? This paper asks whether those chances hang together as a coherent picture of the world. The authors borrow an old idea from probability theory: if your stated chances are inconsistent, someone can bet against you in a way that makes money no matter what happens, a trap called a Dutch book. They turn that trap into a test. They ask models for probabilities about future stock returns falling into different ranges, then use optimization to find the best guaranteed-profit betting scheme against those answers. That guaranteed profit is their score for incoherence, and it needs no observed outcomes. Across 100 stock-days and 15 models they find widespread incoherence, much larger differences in coherence than in accuracy, worse incoherence when events are logically linked, and striking sensitivity to prompt wording — including irrelevant chatter that can raise incoherence tenfold.
The problem they're solving, and why it matters
The problem is trust in model forecasts. Model ensembles can now rival human crowds at forecasting, yet models often contradict themselves. A simple example is saying an event has 70% chance while also saying its opposite has 50% chance, so the two add to more than 100%. For a user making a real decision, that is worrying because we implicitly assume the model has something like an internal world model — a consistent picture of how likely different futures are — from which its answers are read off. If answers do not fit together, there is no such picture. Existing ways to judge forecasts usually need labels, meaning we must wait to see what actually happened and then check accuracy or calibration, which means how well stated chances match observed frequencies. That does not work for long-horizon, rare, or not-yet-resolved questions. The authors want a label-free check that uses only the forecasts themselves plus their known logical structure, and that exhausts all implications of consistency at once rather than checking one rule at a time.
The key idea, in plain words
The key idea is coherence as bettability. Coherence here means the stated chances could have come from some single consistent probability distribution over outcomes. In plain terms: there exists a way to assign chances to the finest-grained possibilities that adds up correctly to everything the model said. A classic result due to Bruno de Finetti says coherence is exactly the same as immunity to a Dutch book. A Dutch book, a term the authors define to mean a set of bets priced at your stated probabilities that earns a positive profit in every possible world, is only possible when you are incoherent. If you are coherent, no such guaranteed win exists. So instead of arguing philosophically about what models believe, they play arbitrageur. Treat each model probability as a price for a ticket that pays one dollar if the event happens. Allow buying, meaning betting the event will happen, or selling, meaning betting it will not. Look for the combination of buys and sells with limited total stake that maximizes the worst-case profit. If the best worst-case profit is zero, the forecasts are coherent. If it is positive, its size measures how incoherent they are. The authors also show this profit equals the distance, in the sense of the largest gap across events, from the stated forecasts to the nearest coherent set of forecasts.
How it actually works, step by step
First they build a testbed where logic is known by construction. They take daily stock returns from a market database and headlines from a news database, focusing on anchor dates from August to December 2025. That window is chosen to fall after the release of their main model, GPT-OSS-120B, to reduce lookahead bias, which means the risk that a model has already seen the future outcome in training. For each stock-day they show the model the prior 60 daily returns, rescaled to have standard deviation one, plus that scale factor, with dates shown but company name hidden by default. The future return to be forecast is also rescaled and sorted into four bins: far down, mildly down, mildly up, and far up, with cutoffs at minus one, zero, and plus one standard deviations. At one-day horizon there are 14 non-trivial events you can form from those bins, such as a single bin, complements like not in a bin, and unions like in either of two bins. For longer horizons of up to four days and for pairs of stocks they add joint events like up on both days or stock A down while stock B is up. Next they elicit forecasts. The baseline asks about one event per independent query, to mimic real deployments where questions arrive separately with no shared context. The model must return only a number between zero and one to four decimal places. They repeat each prompt five times and average the resulting profit scores. They also run variants: adding summary statistics or forecasting instructions, revealing company identity or a pre-close headline, asking about pairs or the full 14-event panel at once, averaging repeated answers, and inserting irrelevant notes like having coffee that morning, feeling optimistic or scared, having savings in the stock, or seeing a phone battery at 70%. Finally they score. For each stock-day they collect the model's numbers for a fixed event set and solve a linear program, which means an optimization with straight-line objectives and constraints that a computer can solve exactly. Conceptually it searches over all betting portfolios with total absolute stake capped at one, and finds the one with highest guaranteed payoff across all finest-grained outcomes, which the paper calls atoms. An atom here means a mutually exclusive detailed outcome the events distinguish, for example a particular combination of bins. The value found is called arbitrage profit. A worked example in the paper shows the model saying 0.40 for one interval, 0.3833 for the neighboring interval, but 0.8167 for their union, which should be the sum. Buying a third of a unit of each piece and selling a third of the union then profits about 0.011 in every world.
What they showed — results, honestly, with limitations
Baseline incoherence is real and not just rounding. For the workhorse model the mean profit is about 0.00207 per unit staked, with uncertainty interval roughly 0.00147 to 0.00274. Almost half of stock-days show profit above 0.001, and six in a hundred show profit above 0.01. Rounding to four decimals could mechanically create at most 0.00005, so the authors rule that out as the explanation. Across 15 models, coherence varies enormously while accuracy varies little. Mean profit ranges from about 0.002 for the best model to about 0.199 for the worst, roughly a factor of 100. Mean Brier score, which means average squared error between the stated chance and whether the event happened, ranges only from about 0.197 to 0.216. More coherent models also tend to be more accurate, with rank correlation about 0.91, but coherence separates models far more sharply: 101 of 105 pairwise profit differences are statistically distinguishable, versus only 32 for accuracy. Richer logic hurts. Adding joint events across two stocks or across days raises profit significantly beyond using marginal events alone, even when the number of events is held fixed in follow-up linked-versus-dispersed comparisons. In plain terms, asking the model to price pieces that must add up to a whole exposes more inconsistency. Prompt details matter a lot. Showing precomputed bin frequencies helps a bit, while prescribing a count-and-adjust procedure hurts a lot. Naming the company hurts a little, adding a headline hurts a lot. Asking about an event and its complement together helps, asking about unrelated pairs hurts, and asking about all 14 events at once drives incoherence nearly to zero. Averaging five repeats helps only modestly, largely because answers are usually identical. Most irrelevant-context insertions significantly raise incoherence, with an intuitive gut anchor around 70% and positive emotion causing the largest jumps — on the order of ten times baseline in some descriptions. Limitations are stated clearly. This is one domain, standardized stock returns, so generalization to other domains is not shown. Most elicitation experiments use one open-weight model, so cross-model ordering under other prompts is not established. Coherence is a property of outputs, not proof about inner beliefs, it depends on which events are queried, and a dogmatic but consistent forecaster can be perfectly coherent yet badly inaccurate. Accuracy comparisons beyond the workhorse model may also be affected by training-data lookahead.
Why this might matter to me
If you care about spatial reasoning, world models, text-to-CAD and parametric geometry, or 3D generation, read coherence as geometric consistency. A world model that says a chair leg has 80% chance of being inside a footprint while also saying it has 40% chance of being outside it has no consistent scene representation, just as a forecaster who violates complementarity has no consistent distribution. Your domains are full of such must-add-up constraints: parts that must partition a solid, sketches whose constraints must jointly solve, occupancy that must sum to one across voxels, or views that must agree on one underlying 3D shape. The label-free aspect is directly useful because ground truth is expensive in geometry: you can check whether predicted chances over spatial bins, contacts, or parametric choices fit together without building the CAD model or rendering the scene. The finding that asking about the full panel at once nearly eliminates incoherence suggests a practical pattern: elicit or train on constraint-closed sets — mating faces, extrude parameters and their complements, multi-view occupancies — rather than isolated queries. Conversely, the result that irrelevant linguistic context inflates incoherence is a warning for text-to-CAD prompts full of stylistic chatter or numeric anchors. Finally, the Dutch-book linear program offers a concrete training signal: an exact set-level penalty for violating additivity, which could complement per-example accuracy rewards when teaching models to respect geometry.
Terms worth knowing
- Coherence: the property that a set of stated probabilities could come from one consistent probability distribution. Example: chances for an event and its opposite sum to one.
- Dutch book: a collection of bets priced at your stated probabilities that yields strictly positive profit no matter which outcome occurs. Its existence proves incoherence.
- Arbitrage profit: here, the largest worst-case profit per unit of total stake an arbitrageur could guarantee by betting for or against the model's probabilities. Zero means coherent; larger means more incoherent.
- Atom: the finest-grained mutually exclusive outcome distinguished by the event set. Example: a specific combination of return bins across days.
- Incidence matrix: a table recording which atoms belong to which events, used to compute payoffs.
- Brier score: average squared difference between stated probability and actual outcome indicator, zero for perfect and larger for worse. Measures accuracy, not coherence.
- Calibration: whether events stated with, say, 70% chance actually happen about 70% of the time. Requires observed outcomes.
- Lookahead bias: the risk that a model's training data already contains the supposedly future event, inflating measured accuracy.
- Elicitation arm: a variant of the prompting protocol, such as adding statistics, grouping questions, or inserting irrelevant context, holding the events fixed to isolate its effect.