Blog
First five resolutions: one benchmark beaten out of four
Five forecasts on this site have now resolved. All five are featured questions, so they are selection-biased by construction and say little about skill on their own. They are still worth reading closely, because the misses are not random.
The numbers
| Question | Forecast | Outcome | Ex-ante benchmark | Beat it? |
|---|---|---|---|---|
| FOMC, 15–16 Sep: upper-bound hike? | 63% yes | Yes | none recorded | — |
| US distillate stocks, week to 25 Sep | median 102.0 Mbbl | 105.18 | persistence 106.3 | No |
| US core PCE m/m, August | median 0.195% | 0.2% | persistence 0.2% | No |
| US initial claims, week to 19 Sep | median 226,500 | 197,000 | 4-week average 203,250 | No |
| USD/JPY, H.10 noon rate, 30 Sep | median 155.25 | 157.25 | persistence 153.71 | Yes |
Against the uninformative baseline the mean log-score difference is about +0.03 per question: barely positive, and with n = 5 indistinguishable from zero. Against the benchmarks that were frozen before each cutoff, the record is one win in four on CRPS. The track-record page shows both, because the second number is the harder and more honest test.
What the misses have in common
Two of the three losses are weekly level series: distillate stocks and jobless claims. On a one- or two-week horizon these series barely move, and the last official print is very hard to beat. Both forecasts moved away from the latest level, towards a story about where the level "should" be. The claims forecast is the clearest case: a median of 226,500 when the series had been running near 200,000. That is a regime error, not bad luck.
The core PCE miss is different. The centre was right (0.195 against an outcome of 0.2), but the official figure is rounded to one decimal, and the benchmark was a point mass exactly on the rounded value. When the outcome lands on the benchmark's single point, no spread-out distribution can beat it on CRPS. The loss is real under the rules, and it is also a reminder that rounded outcomes reward point forecasts.
USD/JPY is the one win. The yen kept weakening through late September, persistence (the 11 September level) was left behind, and the outcome landed at the forecast's upper quartile.
A rule we are keeping, even though it costs us
The USD/JPY outcome, 157.25, fell exactly on a knot of the published CDF. At a knot the density is not defined, and the scoring code (published before any outcome) uses the segment to the right of the point, which here has the lower density. Using the left segment would have turned that question's log-score difference from negative to clearly positive. We are not changing the convention: a scoring rule edited after the outcome, in the direction that helps, is no longer a rule.
What changes
- Benchmarks are now computed and shown on every resolved forecast, next to the uninformative baseline.
- The method can now start from an external, citable observable (a consensus, an official nowcast, a market-implied level) instead of only from the model ensemble. Not every question will use one; which observable qualifies depends on the question.
- New forecasts this week raise the bar on purpose. The US Q3 GDP forecast (median 2.75%) is benchmarked against the Atlanta Fed's GDPNow nowcast, frozen at cutoff at 3.6%, which is a much stronger opponent than persistence. The forecast sits well below the nowcast; if the nowcast is right, that will show up here as a loss. The ECB question (a rate rise on 29 October) is at 28.5%.
A correction about timestamps
Until today, the methodology and sealed-forecast pages said that forecasts were timestamped by entering "public Git history". The repository behind this site is private, so readers could not check that claim. It has been removed.
The timing evidence that a reader can check is now an independent archive. After each change to the ledger, the published forecasts.json is captured by the Internet Archive, the archived bytes are downloaded back, and the capture is listed in /data/archives.json only if its SHA-256 matches the published file. Each forecast card links the earliest capture that holds its current state, and says so plainly when that capture came after the question closed, in which case it is not evidence of timing. For the records so far: the distillate and USD/JPY forecasts, and the first state of the core PCE forecast, were captured before they closed; the FOMC and jobless-claims forecasts, and the second core PCE state, were not.
Update, later on 13 October. Both forecasts above now have a second state. Two of the engine's providers had been unavailable at the morning run; once they were back, both questions were re-run under one rule decided before seeing the results: full provider set, plus an external entry prior (for the ECB, a published poll of economists, 70 of 73 expecting a hold; for GDP, the New York Fed's nowcast, keeping the Atlanta Fed's GDPNow as the independent benchmark). The ECB forecast moved from 28.5% to 7.6%; GDP from a median of 2.75% to 2.47%. The first states stay on the record and in the archived captures; the standing (latest) state is the one scored.
Five questions are not a track record. They are a first look at where this method fails, and we would rather publish that than wait for a nicer sample.