Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System

Haiying Xu; Kevin Paul; Anderson Banihirwe

doi:10.1109/PDSW51947.2020.00012

Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System

Haiying Xu, Kevin Paul, Anderson Banihirwe

Application Scalability and Performance Group

National Center for Atmospheric Research

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Pangeo is a community of scientists and software developers collaborating to enable Big Data Geoscience analysis interactively in the public cloud and on high-performance computing (HPC) systems. At the core of the Pangeo software stack is (1) Xarray, which adds labels to metadata such as dimensions, coordinates and attributes for raw array-oriented data, (2) Dask, which provides parallel computation and out-of-core memory capabilities, and (3) Jupyter Lab which offers the web-based interactive environment to the Pangeo platform. Geoscientists now have a strong candidate software stack to analyze large datasets, and they are very curious about performance differences between the Zarr and NetCDF4 data formats on both traditional file storage systems and object storage. We have written a benchmarking suite for the Pangeo stack that can measure scalability and performance information of both input/output (I/O) throughput and computation. We will describe how we performed these benchmarks, analyzed our results, and we will discuss the pros and cons of the Pangeo software stack in terms of I/O scalability on both cloud and HPC storage systems.

Original language	English
Title of host publication	Proceedings of PDSW 2020
Subtitle of host publication	IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	40-45
Number of pages	6
ISBN (Electronic)	9781665415941
DOIs	https://doi.org/10.1109/PDSW51947.2020.00012
State	Published - Nov 2020
Event	5th IEEE/ACM International Parallel Data Systems Workshop, PDSW 2020 - Virtual, Atlanta, United States Duration: Nov 12 2020 → …

Publication series

Name	Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	5th IEEE/ACM International Parallel Data Systems Workshop, PDSW 2020
Country/Territory	United States
City	Virtual, Atlanta
Period	11/12/20 → …

Keywords

HPC
I/O
Pangeo
benchmark
cloud
object store
throughput

Access to Document

10.1109/PDSW51947.2020.00012

Cite this

Xu, H., Paul, K., & Banihirwe, A. (2020). Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System. In Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 40-45). Article 9306992 (Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/PDSW51947.2020.00012

Xu, Haiying ; Paul, Kevin ; Banihirwe, Anderson. / Pangeo Benchmarking Analysis : Object Storage vs. POSIX File System. Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 40-45 (Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{eac6cde4cf684c8db9d21d3fe0d0b955,

title = "Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System",

abstract = "Pangeo is a community of scientists and software developers collaborating to enable Big Data Geoscience analysis interactively in the public cloud and on high-performance computing (HPC) systems. At the core of the Pangeo software stack is (1) Xarray, which adds labels to metadata such as dimensions, coordinates and attributes for raw array-oriented data, (2) Dask, which provides parallel computation and out-of-core memory capabilities, and (3) Jupyter Lab which offers the web-based interactive environment to the Pangeo platform. Geoscientists now have a strong candidate software stack to analyze large datasets, and they are very curious about performance differences between the Zarr and NetCDF4 data formats on both traditional file storage systems and object storage. We have written a benchmarking suite for the Pangeo stack that can measure scalability and performance information of both input/output (I/O) throughput and computation. We will describe how we performed these benchmarks, analyzed our results, and we will discuss the pros and cons of the Pangeo software stack in terms of I/O scalability on both cloud and HPC storage systems.",

keywords = "HPC, I/O, Pangeo, benchmark, cloud, object store, throughput",

author = "Haiying Xu and Kevin Paul and Anderson Banihirwe",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 5th IEEE/ACM International Parallel Data Systems Workshop, PDSW 2020 ; Conference date: 12-11-2020",

year = "2020",

month = nov,

doi = "10.1109/PDSW51947.2020.00012",

language = "English",

series = "Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "40--45",

booktitle = "Proceedings of PDSW 2020",

address = "United States",

}

Xu, H, Paul, K & Banihirwe, A 2020, Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System. in Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis., 9306992, Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 40-45, 5th IEEE/ACM International Parallel Data Systems Workshop, PDSW 2020, Virtual, Atlanta, United States, 11/12/20. https://doi.org/10.1109/PDSW51947.2020.00012

Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System. / Xu, Haiying; Paul, Kevin; Banihirwe, Anderson.
Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2020. p. 40-45 9306992 (Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Pangeo Benchmarking Analysis

T2 - 5th IEEE/ACM International Parallel Data Systems Workshop, PDSW 2020

AU - Xu, Haiying

AU - Paul, Kevin

AU - Banihirwe, Anderson

PY - 2020/11

Y1 - 2020/11

N2 - Pangeo is a community of scientists and software developers collaborating to enable Big Data Geoscience analysis interactively in the public cloud and on high-performance computing (HPC) systems. At the core of the Pangeo software stack is (1) Xarray, which adds labels to metadata such as dimensions, coordinates and attributes for raw array-oriented data, (2) Dask, which provides parallel computation and out-of-core memory capabilities, and (3) Jupyter Lab which offers the web-based interactive environment to the Pangeo platform. Geoscientists now have a strong candidate software stack to analyze large datasets, and they are very curious about performance differences between the Zarr and NetCDF4 data formats on both traditional file storage systems and object storage. We have written a benchmarking suite for the Pangeo stack that can measure scalability and performance information of both input/output (I/O) throughput and computation. We will describe how we performed these benchmarks, analyzed our results, and we will discuss the pros and cons of the Pangeo software stack in terms of I/O scalability on both cloud and HPC storage systems.

AB - Pangeo is a community of scientists and software developers collaborating to enable Big Data Geoscience analysis interactively in the public cloud and on high-performance computing (HPC) systems. At the core of the Pangeo software stack is (1) Xarray, which adds labels to metadata such as dimensions, coordinates and attributes for raw array-oriented data, (2) Dask, which provides parallel computation and out-of-core memory capabilities, and (3) Jupyter Lab which offers the web-based interactive environment to the Pangeo platform. Geoscientists now have a strong candidate software stack to analyze large datasets, and they are very curious about performance differences between the Zarr and NetCDF4 data formats on both traditional file storage systems and object storage. We have written a benchmarking suite for the Pangeo stack that can measure scalability and performance information of both input/output (I/O) throughput and computation. We will describe how we performed these benchmarks, analyzed our results, and we will discuss the pros and cons of the Pangeo software stack in terms of I/O scalability on both cloud and HPC storage systems.

KW - HPC

KW - I/O

KW - Pangeo

KW - benchmark

KW - cloud

KW - object store

KW - throughput

UR - https://www.scopus.com/pages/publications/85099553692

U2 - 10.1109/PDSW51947.2020.00012

DO - 10.1109/PDSW51947.2020.00012

M3 - Conference contribution

AN - SCOPUS:85099553692

T3 - Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 40

EP - 45

BT - Proceedings of PDSW 2020

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 12 November 2020

ER -

Xu H, Paul K, Banihirwe A. Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System. In Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2020. p. 40-45. 9306992. (Proceedings of PDSW 2020: IEEE/ACM 5th International Parallel Data Systems Workshop, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/PDSW51947.2020.00012

Pangeo Benchmarking Analysis: Object Storage vs. POSIX File System

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this