Light-weight parallel Python tools for earth system modeling workflows

Kevin Paul; Sheri Mickelson; John M. Dennis; Haiying Xu; David Brown

doi:10.1109/BigData.2015.7363979

Light-weight parallel Python tools for earth system modeling workflows

Kevin Paul, Sheri Mickelson, John M. Dennis, Haiying Xu, David Brown

Application Scalability and Performance Group

National Center for Atmospheric Research

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

12 Scopus citations

Abstract

In the last 30 years, earth system modeling has become increasingly data-intensive. The Community Earth System Model (CESM) response to the next Intergovernmental Panel on Climate Change (IPCC) assessment report (AR6) may require close to 1 Billion CPU hours of computation and generate up to 12 PB of raw data for post-processing. Existing post-processing tools are serial-only and impossibly slow with this much data. To improve the post-processing performance, our team has adopted a strategy of targeted replacement of the «bottleneck software» with light-weight parallel Python alternatives. This allows maximum impact with the least disruption to the CESM community and the shortest delivery time. We developed two light-weight parallel Python tools: one to convert model output from time-slice to time-series format, and one to perform fast time-averaging of time-series data. We present the motivation, approach, and results of these two tools, and our plans for future research and development.

Original language	English
Title of host publication	Proceedings - 2015 IEEE International Conference on Big Data, Big Data 2015
Editors	Howard Ho, Beng Chin Ooi, Mohammed J. Zaki, Xiaohua Hu, Laura Haas, Vipin Kumar, Sudarsan Rachuri, Shipeng Yu, Morris Hui-I Hsiao, Jian Li, Feng Luo, Saumyadipta Pyne, Kemafor Ogan
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1985-1994
Number of pages	10
ISBN (Electronic)	9781479999255
DOIs	https://doi.org/10.1109/BigData.2015.7363979
State	Published - Dec 22 2015
Event	3rd IEEE International Conference on Big Data, Big Data 2015 - Santa Clara, United States Duration: Oct 29 2015 → Nov 1 2015

Publication series

Name	Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015

Conference

Conference	3rd IEEE International Conference on Big Data, Big Data 2015
Country/Territory	United States
City	Santa Clara
Period	10/29/15 → 11/1/15

Keywords

Data processing
High performance computing
Parallel processing
Scientific computing

Access to Document

10.1109/BigData.2015.7363979

Cite this

Paul, K., Mickelson, S., Dennis, J. M., Xu, H., & Brown, D. (2015). Light-weight parallel Python tools for earth system modeling workflows. In H. Ho, B. C. Ooi, M. J. Zaki, X. Hu, L. Haas, V. Kumar, S. Rachuri, S. Yu, M. H.-I. Hsiao, J. Li, F. Luo, S. Pyne, & K. Ogan (Eds.), Proceedings - 2015 IEEE International Conference on Big Data, Big Data 2015 (pp. 1985-1994). Article 7363979 (Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData.2015.7363979

Paul, Kevin ; Mickelson, Sheri ; Dennis, John M. et al. / Light-weight parallel Python tools for earth system modeling workflows. Proceedings - 2015 IEEE International Conference on Big Data, Big Data 2015. editor / Howard Ho ; Beng Chin Ooi ; Mohammed J. Zaki ; Xiaohua Hu ; Laura Haas ; Vipin Kumar ; Sudarsan Rachuri ; Shipeng Yu ; Morris Hui-I Hsiao ; Jian Li ; Feng Luo ; Saumyadipta Pyne ; Kemafor Ogan. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 1985-1994 (Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015).

@inproceedings{174c618b3077497dba94c6484b266f5c,

title = "Light-weight parallel Python tools for earth system modeling workflows",

abstract = "In the last 30 years, earth system modeling has become increasingly data-intensive. The Community Earth System Model (CESM) response to the next Intergovernmental Panel on Climate Change (IPCC) assessment report (AR6) may require close to 1 Billion CPU hours of computation and generate up to 12 PB of raw data for post-processing. Existing post-processing tools are serial-only and impossibly slow with this much data. To improve the post-processing performance, our team has adopted a strategy of targeted replacement of the «bottleneck software» with light-weight parallel Python alternatives. This allows maximum impact with the least disruption to the CESM community and the shortest delivery time. We developed two light-weight parallel Python tools: one to convert model output from time-slice to time-series format, and one to perform fast time-averaging of time-series data. We present the motivation, approach, and results of these two tools, and our plans for future research and development.",

keywords = "Data processing, High performance computing, Parallel processing, Scientific computing",

author = "Kevin Paul and Sheri Mickelson and Dennis, \{John M.\} and Haiying Xu and David Brown",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 3rd IEEE International Conference on Big Data, Big Data 2015 ; Conference date: 29-10-2015 Through 01-11-2015",

year = "2015",

month = dec,

day = "22",

doi = "10.1109/BigData.2015.7363979",

language = "English",

series = "Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1985--1994",

editor = "Howard Ho and Ooi, \{Beng Chin\} and Zaki, \{Mohammed J.\} and Xiaohua Hu and Laura Haas and Vipin Kumar and Sudarsan Rachuri and Shipeng Yu and Hsiao, \{Morris Hui-I\} and Jian Li and Feng Luo and Saumyadipta Pyne and Kemafor Ogan",

booktitle = "Proceedings - 2015 IEEE International Conference on Big Data, Big Data 2015",

address = "United States",

}

Paul, K, Mickelson, S , Dennis, JM , Xu, H & Brown, D 2015, Light-weight parallel Python tools for earth system modeling workflows. in H Ho, BC Ooi, MJ Zaki, X Hu, L Haas, V Kumar, S Rachuri, S Yu, MH-I Hsiao, J Li, F Luo, S Pyne & K Ogan (eds), Proceedings - 2015 IEEE International Conference on Big Data, Big Data 2015., 7363979, Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015, Institute of Electrical and Electronics Engineers Inc., pp. 1985-1994, 3rd IEEE International Conference on Big Data, Big Data 2015, Santa Clara, United States, 10/29/15. https://doi.org/10.1109/BigData.2015.7363979

Light-weight parallel Python tools for earth system modeling workflows. / Paul, Kevin; Mickelson, Sheri ; Dennis, John M. et al.
Proceedings - 2015 IEEE International Conference on Big Data, Big Data 2015. ed. / Howard Ho; Beng Chin Ooi; Mohammed J. Zaki; Xiaohua Hu; Laura Haas; Vipin Kumar; Sudarsan Rachuri; Shipeng Yu; Morris Hui-I Hsiao; Jian Li; Feng Luo; Saumyadipta Pyne; Kemafor Ogan. Institute of Electrical and Electronics Engineers Inc., 2015. p. 1985-1994 7363979 (Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Light-weight parallel Python tools for earth system modeling workflows

AU - Paul, Kevin

AU - Mickelson, Sheri

AU - Dennis, John M.

AU - Xu, Haiying

AU - Brown, David

PY - 2015/12/22

Y1 - 2015/12/22

N2 - In the last 30 years, earth system modeling has become increasingly data-intensive. The Community Earth System Model (CESM) response to the next Intergovernmental Panel on Climate Change (IPCC) assessment report (AR6) may require close to 1 Billion CPU hours of computation and generate up to 12 PB of raw data for post-processing. Existing post-processing tools are serial-only and impossibly slow with this much data. To improve the post-processing performance, our team has adopted a strategy of targeted replacement of the «bottleneck software» with light-weight parallel Python alternatives. This allows maximum impact with the least disruption to the CESM community and the shortest delivery time. We developed two light-weight parallel Python tools: one to convert model output from time-slice to time-series format, and one to perform fast time-averaging of time-series data. We present the motivation, approach, and results of these two tools, and our plans for future research and development.

AB - In the last 30 years, earth system modeling has become increasingly data-intensive. The Community Earth System Model (CESM) response to the next Intergovernmental Panel on Climate Change (IPCC) assessment report (AR6) may require close to 1 Billion CPU hours of computation and generate up to 12 PB of raw data for post-processing. Existing post-processing tools are serial-only and impossibly slow with this much data. To improve the post-processing performance, our team has adopted a strategy of targeted replacement of the «bottleneck software» with light-weight parallel Python alternatives. This allows maximum impact with the least disruption to the CESM community and the shortest delivery time. We developed two light-weight parallel Python tools: one to convert model output from time-slice to time-series format, and one to perform fast time-averaging of time-series data. We present the motivation, approach, and results of these two tools, and our plans for future research and development.

KW - Data processing

KW - High performance computing

KW - Parallel processing

KW - Scientific computing

UR - https://www.scopus.com/pages/publications/84963774050

U2 - 10.1109/BigData.2015.7363979

DO - 10.1109/BigData.2015.7363979

M3 - Conference contribution

AN - SCOPUS:84963774050

T3 - Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015

SP - 1985

EP - 1994

BT - Proceedings - 2015 IEEE International Conference on Big Data, Big Data 2015

A2 - Ho, Howard

A2 - Ooi, Beng Chin

A2 - Zaki, Mohammed J.

A2 - Hu, Xiaohua

A2 - Haas, Laura

A2 - Kumar, Vipin

A2 - Rachuri, Sudarsan

A2 - Yu, Shipeng

A2 - Hsiao, Morris Hui-I

A2 - Li, Jian

A2 - Luo, Feng

A2 - Pyne, Saumyadipta

A2 - Ogan, Kemafor

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 3rd IEEE International Conference on Big Data, Big Data 2015

Y2 - 29 October 2015 through 1 November 2015

ER -

Paul K, Mickelson S , Dennis JM , Xu H, Brown D. Light-weight parallel Python tools for earth system modeling workflows. In Ho H, Ooi BC, Zaki MJ, Hu X, Haas L, Kumar V, Rachuri S, Yu S, Hsiao MHI, Li J, Luo F, Pyne S, Ogan K, editors, Proceedings - 2015 IEEE International Conference on Big Data, Big Data 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 1985-1994. 7363979. (Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015). doi: 10.1109/BigData.2015.7363979

Light-weight parallel Python tools for earth system modeling workflows

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this