TY - GEN
T1 - What to Support When You're Compressing The State of Practice Gaps and Opportunities for Scientific Data Compression
AU - Cappello, Franck
AU - Underwood, Robert
AU - Alexeev, Yuri
AU - Baker, Alison
AU - Bozdağ, Ebru
AU - Burtscher, Martin
AU - Chard, Kyle
AU - Di, Sheng
AU - Felker, Kyle Gerard
AU - O'Grady, Paul Christopher
AU - Guo, Hanqi
AU - Huang, Yafan
AU - Jiang, Peng
AU - Jin, Sian
AU - Johansson, Petter
AU - Li, Shaomeng
AU - Liang, Xin
AU - Lindahl, Erik
AU - Lindstrom, Peter
AU - Lukić, Zarija
AU - Lundborg, Magnus
AU - Lykov, Danylo
AU - Nagaso, Masaru
AU - Sato, Kento
AU - Singh, Amarjit
AU - Son, Seung Woo
AU - Song, Shihui
AU - Tang, William
AU - Tao, Dingwen
AU - Tian, Jiannan
AU - Yoshii, Kazutomo
AU - Zhao, Kai
N1 - Publisher Copyright:
© 2025 Copyright held by the owner/author(s).
PY - 2025/11/15
Y1 - 2025/11/15
N2 - Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications' data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications' needs and compressors' capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.
AB - Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications' data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications' needs and compressors' capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.
KW - Error-Bounded Lossy Compression
KW - High-Performance Computing
KW - I/O Optimization
KW - Requirements Analysis
KW - State of Practice
UR - https://www.scopus.com/pages/publications/105023972839
U2 - 10.1145/3712285.3759856
DO - 10.1145/3712285.3759856
M3 - Conference contribution
AN - SCOPUS:105023972839
T3 - Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025
SP - 1966
EP - 1979
BT - Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025
PB - Association for Computing Machinery, Inc
T2 - 2025 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2025
Y2 - 16 November 2025 through 21 November 2025
ER -