Implementation and Assessment of METplus Verification for the NSSL Experimental Warn-on-Forecast System

Verification of convection-allowing models (CAM) plays an instrumental role in the development and improvement of model-based performance. This year, a near real-time workflow was developed to perform verification of the National Severe Storms Laboratory (NSSL) Warn-on-Forecast System (WoFS) forecast output using the community-based, enhanced Model Evaluation Tools (METplus) verification suite along with a companion database and visualization tool (METviewer). The WoFS aims to provide a continuous flow of short-term (0-6 hr) probabilistic guidance of hazardous weather threats such as flash flooding. In collaboration with operational forecasters at the Weather Prediction Center's (WPC) MetWatch Desk, WoFS runs were completed in real-time over domains of interest based on forecasted flash flooding risks denoted by the Weather Prediction Center's (WPC) Day 1 Excessive Rainfall Outlook.

To facilitate a comparison of WoFS performance relative to other CAMs, output was also obtained from the High-Resolution Ensemble Forecast (HREF) as well as a time-lagged version of the High-Resolution Rapid Refresh (HRRR-TLE). All model data were verified against observations from the NSSL's operational Multi-Radar/Multi-Sensor System (MRMS). Specifically, accumulated precipitation and composite reflectivity from each ensemble are extracted and are compared to the corresponding MRMS observations that have been interpolated to the WoFS model grid. Fixed thresholds for accumulated rainfall and composite reflectivity were utilized during the real-time runs while percentile thresholds were calculated and applied retrospectively to provide insight into individual model climatologies. A neighborhood maximum ensemble probability technique (NMEP; Schwartz and Sobash 2017) was applied to the probabilistic forecasts with 3, 15, and 27 km neighborhoods for each ensemble and verification metrics were computed using METplus on the smoothed probability fields. Using METviewer, a scorecard was generated to highlight statistically significant differences between WoFS and the HREF and HRRR-TLE. The automated production of these scorecards allowed for a rapid interpretation of WoFS performance for a particular day. Results from this experiment will be presented to discuss key strengths and potential improvement areas for WoFS.