Examining Variations in the Optimal Compression Level of Spatiotemporal Datasets Determined Using the Data Structural Similarity Index Measure (DSSIM)

Lossy compression of climate model output is desperately needed to reduce the massive storage burden on research centers. Lossy compression must be applied carefully to avoid inadvertently affecting any scientific analyses. At a minimum, the compressed datasets should not be visually different when compared with the original datasets. The DSSIM is a promising metric that is effective as classifying images as visually distinct or indistinct. Here we use the DSSIM to determine the optimal compression levels for climate datasets and then examine the behavior of the DSSIM applied to CESM-LENS data under multiple compression levels. We note that the DSSIM is not monotonic in the compression level and examine this behavior in more depth. We also obtain the optimal compression levels for time slices of each variable to get a feel for how these levels vary, and compute the optimal compression level for every variable. Finally, we look at how the optimal compression level obtained using the DSSIM is related to the file size of each compressed dataset.