Hybrid ensemble-variational filter: A spatially and temporally varying adaptive algorithm to estimate relative weighting

Model errors and sampling errors produce inaccurate sample covariances that limit the performance of ensemble Kalman filters. Linearly hybridizing the flow-dependent ensemble-based covariance with a time-invariant background covariance matrix gives a better estimate of the true error covariance. Previous studies have shown this, both in theory and in practice. How to choose the weight for each covariance remains an open question especially in the presence of model biases. This study assumes the weighting coefficient to be a random variable and then introduces a Bayesian scheme to estimate it using the available data. The scheme takes into account the discrepancy between the ensemble mean and the observations, the ensemble variance, the static background variance, and the uncertainties in the observations. The proposed algorithm is first derived for a spatially constant weight and then this assumption is relaxed by estimating a unique scalar weight for each state variable. Using twin experiments with the 40-variable Lorenz 96 system, it is shown that the proposed scheme is able to produce quality forecasts even in the presence of severe sampling errors. The adaptive algorithm allows the hybrid filter to switch between an EnKF and a simple EnOI depending on the statistics of the ensemble. In the presence of model errors, the adaptive scheme demonstrates additional improvements compared with standard enhancements alone, such as inflation and localization. Finally, the potential of the spatially varying variant to accommodate challenging sparse observation networks is demonstrated. The computational efficiency and storage of the proposed scheme, which remain an obstacle, are discussed.