Verification of prognostic products

Quality Control of forecast products through the unified system named VERSUS (VERification System Unified Survey) based on RDBMS system and a web-based GUI are carried out routinely at COMET. The VERSUS code has been developed from the Italian Air Force Met Service in the framework of the COSMO-VERSUS priority project. The VERSUS software is an unified, flexible and customizable tool, able to perform operational standard verifications, operational conditional verifications along with experimental standard and conditional verifications, in batch and interactive mode. VERSUS performs a systematic evaluation of model performances in order to reveal, in a way different from the usual classical verification tools, model weaknesses. It is able to provide information providing hints which could be the causes of model deficiencies that can usually be seen in the operational verification. The Conditional Verification tool in VERSUS consist of the selection of one or several forecast products and one or several mask variables or conditions, which would be used to define for example thresholds for the product verification (e.g. verification of T2M only for grid points with zero cloud cover in model and observations). Statistical scores and scores plots are routinely produced at COMET for the following NWP models, in operation at COMET: COSMO-ME, for both surface and upper air parameters; COSMO-IT and ECMWF, only for surface parameters.

The basic verified weather parameters for all the models are:

• 2m temperature;
• 10m wind-speed and direction;
• 6h,12h,24h cumulated precipitation;
• MSLP;
• 2m dew point temperature;
• total cloud cover.

along with forecasted upper air parameters:

• temperature;
• geopotential;
• wind-speed and direction;
• relative humidity and/or dew point.

These datasets are regularly available every 3 hours (6 or 12 hours for upper-air) for all the Italian Meteorological stations. A set of stratifications in time are routinely performed as follows:

• Monthly mean
• Quarterly mean
• Yearly mean (mainly for the calculation of a Global quality index)

The scores can be also morphologically (or geographically) stratified in coastal, valley and mountain stations or Northern, Central and Southern Italy, all over Italy and so on.
The scores produced for continuous variable are: Mean Error, Mean Absolute Error, Root Mean Squared Error and their respective Skill Scores for accuracy. As reference scores both persistence, random chance and climatology can be used.
Scores produced for categorical events (e.g. precipitation) are computed from a 2x2 contingency table showing the frequency of “yes” and “no” forecasts and corresponding observations. These contingency tables can be computed for different threshold values. There are a quite large number of measures that can come out from such a table and all of them can be computed through the use of VERSUS software. Among these are frequency bias, probability of detection, false alarm ratio, equitable threat score, and odds ratio.