A Statistical Investigation of the CESM Ensemble Consistency Testing Framework - Part II

Weather is the archetype for chaotic dynamical system. As such, improvements, updates, and modifications to large scale climate simulation demand there be quality checks in place to ensure that new climate simulations are consistent in their reproduction of the real-world phenomena. The Community Earth System Model (CESM) Ensemble Consistency Test (ECT) was constructed to fulfill this need. In short, the CESM-ECT uses a large ensemble of climate simulations produced from an established software version and trusted computational environment as a baseline to compare against new climate simulations produced from updated software and/or a new computational infrastructure. One shortcoming of the current version of CESM-ECT is its need for a large ensemble in order to produce accurate enough estimation for certain statistics. The current method uses Principal Component Analysis (PCA) and in previous work we demonstrate the limitations of this approach for reducing the necessary ensemble size. In this work, we propose an alternative method based on a hard thresholding estimate of the correlation matrix. We compare our new method to the previously established method by considering both the false positive rate and the power in our analysis. We are able to explain a remarkable and counterintuitive result concerning the effect of ensemble size on the power of the CESM-ECT by investigating the effects of ensemble size on the estimates of the eigenvalues of the corresponding sample correlation matrix.