An application-level parallel I/O library for Earth system models

John M. Dennis; Jim Edwards; Ray Loy; Robert Jacob; Arthur A. Mirin; Anthony P. Craig; Mariana Vertenstein

doi:10.1177/1094342011428143

An application-level parallel I/O library for Earth system models

John M. Dennis
, Jim Edwards
, Ray Loy
, Robert Jacob
, Arthur A. Mirin
, Anthony P. Craig
, Mariana Vertenstein

Argonne National Laboratory
Lawrence Livermore Natl. Laboratory
National Center for Atmospheric Research

Research output: Contribution to journal › Article › peer-review

40 Scopus citations

Abstract

We describe the design and implementation of an application-level parallel I/O (PIO) library for the reading and writing of distributed arrays to several common scientific data formats. PIO provides the flexibility to control the number of I/O tasks through data rearrangement to an I/O friendly decomposition. This flexibility enables reductions in per task memory usage and improvements in disk I/O performance versus a serial I/O approach. We illustrate the impact various features within PIO have on memory usage and disk I/O bandwidth on a Cray XT5 system.

Original language	English
Pages (from-to)	43-53
Number of pages	11
Journal	International Journal of High Performance Computing Applications
Volume	26
Issue number	1
DOIs	https://doi.org/10.1177/1094342011428143
State	Published - Feb 2012

Keywords

Earth System Modeling
memory efficient
netCDF
parallel computing
Parallel I/O

Access to Document

10.1177/1094342011428143

Cite this

@article{9e0fe55a48cb486098c4431a0b480073,

title = "An application-level parallel I/O library for Earth system models",

abstract = "We describe the design and implementation of an application-level parallel I/O (PIO) library for the reading and writing of distributed arrays to several common scientific data formats. PIO provides the flexibility to control the number of I/O tasks through data rearrangement to an I/O friendly decomposition. This flexibility enables reductions in per task memory usage and improvements in disk I/O performance versus a serial I/O approach. We illustrate the impact various features within PIO have on memory usage and disk I/O bandwidth on a Cray XT5 system.",

keywords = "Earth System Modeling, memory efficient, netCDF, parallel computing, Parallel I/O",

author = "Dennis, \{John M.\} and Jim Edwards and Ray Loy and Robert Jacob and Mirin, \{Arthur A.\} and Craig, \{Anthony P.\} and Mariana Vertenstein",

year = "2012",

month = feb,

doi = "10.1177/1094342011428143",

language = "English",

volume = "26",

pages = "43--53",

journal = "International Journal of High Performance Computing Applications",

issn = "1094-3420",

publisher = "SAGE Publications Inc.",

number = "1",

}

TY - JOUR

T1 - An application-level parallel I/O library for Earth system models

AU - Dennis, John M.

AU - Edwards, Jim

AU - Loy, Ray

AU - Jacob, Robert

AU - Mirin, Arthur A.

AU - Craig, Anthony P.

AU - Vertenstein, Mariana

PY - 2012/2

Y1 - 2012/2

N2 - We describe the design and implementation of an application-level parallel I/O (PIO) library for the reading and writing of distributed arrays to several common scientific data formats. PIO provides the flexibility to control the number of I/O tasks through data rearrangement to an I/O friendly decomposition. This flexibility enables reductions in per task memory usage and improvements in disk I/O performance versus a serial I/O approach. We illustrate the impact various features within PIO have on memory usage and disk I/O bandwidth on a Cray XT5 system.

AB - We describe the design and implementation of an application-level parallel I/O (PIO) library for the reading and writing of distributed arrays to several common scientific data formats. PIO provides the flexibility to control the number of I/O tasks through data rearrangement to an I/O friendly decomposition. This flexibility enables reductions in per task memory usage and improvements in disk I/O performance versus a serial I/O approach. We illustrate the impact various features within PIO have on memory usage and disk I/O bandwidth on a Cray XT5 system.

KW - Earth System Modeling

KW - memory efficient

KW - netCDF

KW - parallel computing

KW - Parallel I/O

UR - https://www.scopus.com/pages/publications/84857764170

U2 - 10.1177/1094342011428143

DO - 10.1177/1094342011428143

M3 - Article

AN - SCOPUS:84857764170

SN - 1094-3420

VL - 26

SP - 43

EP - 53

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

IS - 1

ER -

An application-level parallel I/O library for Earth system models

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this