Which model was closest to what happened?
Twelve forecasts of 2 m temperature for 497 major cities, checked against the temperature reported by the nearest airport (within 30 km) at 00, 06, 12 and 18Z. The ECMWF and DeepMind models are scored on identical cases; each NOAA run adds its error to those cases when NOAA published it, so its n can be smaller. Lower is better.
| Day 1 | Day 2 | Day 3 | Day 5 | Day 7 | Day 10 | |
|---|---|---|---|---|---|---|
| ECMWF and Google DeepMind | ||||||
| AIFS | 1.43n 42635 | 1.49n 42917 | 1.54n 42932 | 1.73n 41663 | 2.19n 41629 | 2.85n 40667 |
| HRES | 1.54n 42635 | 1.64n 42917 | 1.72n 42932 | 1.93n 41663 | 2.33n 41629 | 2.98n 40667 |
| WN2 | 1.45n 42635 | 1.51n 42917 | 1.55n 42932 | 1.72n 41663 | 2.05n 41629 | 2.47n 40667 |
| ENS mean | 1.51n 42635 | 1.59n 42917 | 1.65n 42932 | 1.84n 41663 | 2.12n 41629 | 2.49n 40667 |
| Run by NOAA, from the ECMWF (IFS) and NOAA (GFS) analyses | ||||||
| GraphCast · IFS start | 1.45n 40869 | 1.51n 41179 | 1.57n 41407 | 1.78n 39902 | 2.27n 38978 | 2.89n 38008 |
| GraphCast · GFS start | 1.63n 31264 | 1.67n 31715 | 1.71n 30648 | 1.91n 28982 | 2.30n 28752 | 3.04n 28354 |
| Aurora · IFS start | 1.46n 40031 | 1.54n 40281 | 1.59n 40284 | 1.78n 39252 | 2.11n 38115 | 2.59n 37377 |
| Aurora · GFS start | 1.60n 31264 | 1.65n 31715 | 1.71n 30648 | 1.94n 28982 | 2.30n 28752 | 2.78n 28354 |
| FourCastNet v2 · IFS start | 1.65n 42635 | 1.72n 42917 | 1.78n 42932 | 1.98n 41663 | 2.32n 41629 | 2.95n 40667 |
| FourCastNet v2 · GFS start | 1.73n 31264 | 1.73n 31715 | 1.78n 30648 | 1.99n 28982 | 2.32n 28752 | 2.97n 28354 |
| Pangu-Weather · IFS start | 1.53n 42202 | 1.62n 41161 | 1.68n 41173 | 1.93n 39897 | 2.44n 38995 | 3.02n 38255 |
| Pangu-Weather · GFS start | 1.65n 31264 | 1.69n 31715 | 1.74n 30648 | 1.99n 28982 | 2.41n 28752 | 3.04n 28354 |
| Day 1 | Day 2 | Day 3 | Day 5 | Day 7 | Day 10 | |
|---|---|---|---|---|---|---|
| ECMWF and Google DeepMind | ||||||
| AIFS | 1.76n 42635 | 1.83n 42917 | 1.88n 42932 | 2.06n 41663 | 2.46n 41629 | 3.10n 40667 |
| HRES | 1.87n 42635 | 1.98n 42917 | 2.07n 42932 | 2.27n 41663 | 2.63n 41629 | 3.22n 40667 |
| WN2 | 1.84n 42635 | 1.91n 42917 | 1.96n 42932 | 2.09n 41663 | 2.38n 41629 | 2.77n 40667 |
| ENS mean | 1.89n 42635 | 1.99n 42917 | 2.06n 42932 | 2.22n 41663 | 2.47n 41629 | 2.80n 40667 |
| Run by NOAA, from the ECMWF (IFS) and NOAA (GFS) analyses | ||||||
| GraphCast · IFS start | 1.82n 40869 | 1.90n 41179 | 1.96n 41407 | 2.15n 39902 | 2.60n 38978 | 3.19n 38008 |
| GraphCast · GFS start | 2.03n 31264 | 2.08n 31715 | 2.13n 30648 | 2.31n 28982 | 2.67n 28752 | 3.40n 28354 |
| Aurora · IFS start | 1.86n 40031 | 1.95n 40281 | 2.03n 40284 | 2.23n 39252 | 2.53n 38115 | 3.00n 37377 |
| Aurora · GFS start | 2.06n 31264 | 2.16n 31715 | 2.26n 30648 | 2.52n 28982 | 2.83n 28752 | 3.27n 28354 |
| FourCastNet v2 · IFS start | 1.96n 42635 | 2.05n 42917 | 2.13n 42932 | 2.34n 41663 | 2.66n 41629 | 3.26n 40667 |
| FourCastNet v2 · GFS start | 2.11n 31264 | 2.14n 31715 | 2.19n 30648 | 2.40n 28982 | 2.70n 28752 | 3.33n 28354 |
| Pangu-Weather · IFS start | 1.81n 42202 | 1.92n 41161 | 2.00n 41173 | 2.25n 39897 | 2.73n 38995 | 3.30n 38255 |
| Pangu-Weather · GFS start | 2.03n 31264 | 2.09n 31715 | 2.15n 30648 | 2.37n 28982 | 2.77n 28752 | 3.41n 28354 |
| Day 1 | Day 2 | Day 3 | Day 5 | Day 7 | Day 10 | |
|---|---|---|---|---|---|---|
| ECMWF and Google DeepMind | ||||||
| AIFS | -0.44n 42635 | -0.49n 42917 | -0.53n 42932 | -0.54n 41663 | -0.54n 41629 | -0.50n 40667 |
| HRES | -0.48n 42635 | -0.54n 42917 | -0.58n 42932 | -0.58n 41663 | -0.57n 41629 | -0.56n 40667 |
| WN2 | -0.65n 42635 | -0.68n 42917 | -0.69n 42932 | -0.65n 41663 | -0.61n 41629 | -0.62n 40667 |
| ENS mean | -0.64n 42635 | -0.68n 42917 | -0.71n 42932 | -0.71n 41663 | -0.71n 41629 | -0.71n 40667 |
| Run by NOAA, from the ECMWF (IFS) and NOAA (GFS) analyses | ||||||
| GraphCast · IFS start | -0.59n 40869 | -0.62n 41179 | -0.64n 41407 | -0.62n 39902 | -0.54n 38978 | -0.62n 38008 |
| GraphCast · GFS start | -0.56n 31264 | -0.70n 31715 | -0.76n 30648 | -0.78n 28982 | -0.76n 28752 | -0.87n 28354 |
| Aurora · IFS start | -0.67n 40031 | -0.70n 40281 | -0.74n 40284 | -0.77n 39252 | -0.63n 38115 | -0.36n 37377 |
| Aurora · GFS start | -0.71n 31264 | -0.89n 31715 | -1.00n 30648 | -1.14n 28982 | -1.14n 28752 | -1.21n 28354 |
| FourCastNet v2 · IFS start | -0.44n 42635 | -0.47n 42917 | -0.52n 42932 | -0.60n 41663 | -0.63n 41629 | -0.63n 40667 |
| FourCastNet v2 · GFS start | -0.53n 31264 | -0.63n 31715 | -0.67n 30648 | -0.75n 28982 | -0.76n 28752 | -0.83n 28354 |
| Pangu-Weather · IFS start | -0.28n 42202 | -0.30n 41161 | -0.32n 41173 | -0.31n 39897 | -0.39n 38995 | -0.41n 38255 |
| Pangu-Weather · GFS start | -0.40n 31264 | -0.43n 31715 | -0.47n 30648 | -0.50n 28982 | -0.49n 28752 | -0.62n 28354 |
How much has been scored
Day N needs runs that are at least N days old, so the right-hand columns fill in over the first two weeks. Observations have been archived since 16 Sep 06Z.
What this can and cannot tell you
An airport is not a city, and a grid box is not an airport. The comparison is fair between models because every model is scored on exactly the same cases, but no single number here is "the error of the forecast for that city".
The forecasts are rounded to whole degrees before they reach this site, which adds about 0.3 °C to every RMSE equally. Thirty days is a short sample: a difference of a few hundredths between two models is noise.
The ensemble mean usually scores best at longer lead times. That is expected, not a finding: averaging many forecasts removes the detail that cannot be predicted, which lowers squared error. Its spread is what the Disagreement Index is measured against.