The scaling of many-task computing approaches in python on cluster supercomputers

Monte Lunacek; Jazcek Braden; Thomas Hauser

doi:10.1109/CLUSTER.2013.6702678

The scaling of many-task computing approaches in python on cluster supercomputers

Monte Lunacek, Jazcek Braden, Thomas Hauser

University of Colorado Boulder

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

12 Scopus citations

Abstract

We compare two packages for performing many-task computing (MTC) in Python: IPython Parallel and Celery. We describe these packages in detail and compare their features as applied to many-task computing on a cluster, including a scaling study using over 12,000 cores and several thousand tasks. We use mpi4py as a baseline for our comparisons. Our results suggest that Python is an excellent way to manage many-task computing and that no single technique is the obvious choice in every situation.

Original language	English
Title of host publication	2013 IEEE International Conference on Cluster Computing, CLUSTER 2013
DOIs	https://doi.org/10.1109/CLUSTER.2013.6702678
State	Published - 2013
Event	15th IEEE International Conference on Cluster Computing, CLUSTER 2013 - Indianapolis, IN, United States Duration: Sep 23 2013 → Sep 27 2013

Publication series

Name	Proceedings - IEEE International Conference on Cluster Computing, ICCC
ISSN (Print)	1552-5244

Conference

Conference	15th IEEE International Conference on Cluster Computing, CLUSTER 2013
Country/Territory	United States
City	Indianapolis, IN
Period	09/23/13 → 09/27/13

Access to Document

10.1109/CLUSTER.2013.6702678

Cite this

@inproceedings{2ff4b401380b4672be03a9a445a45de7,

title = "The scaling of many-task computing approaches in python on cluster supercomputers",

abstract = "We compare two packages for performing many-task computing (MTC) in Python: IPython Parallel and Celery. We describe these packages in detail and compare their features as applied to many-task computing on a cluster, including a scaling study using over 12,000 cores and several thousand tasks. We use mpi4py as a baseline for our comparisons. Our results suggest that Python is an excellent way to manage many-task computing and that no single technique is the obvious choice in every situation.",

author = "Monte Lunacek and Jazcek Braden and Thomas Hauser",

year = "2013",

doi = "10.1109/CLUSTER.2013.6702678",

language = "English",

isbn = "9781479908981",

series = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",

booktitle = "2013 IEEE International Conference on Cluster Computing, CLUSTER 2013",

note = "15th IEEE International Conference on Cluster Computing, CLUSTER 2013 ; Conference date: 23-09-2013 Through 27-09-2013",

}

Lunacek, M, Braden, J & Hauser, T 2013, The scaling of many-task computing approaches in python on cluster supercomputers. in 2013 IEEE International Conference on Cluster Computing, CLUSTER 2013., 6702678, Proceedings - IEEE International Conference on Cluster Computing, ICCC, 15th IEEE International Conference on Cluster Computing, CLUSTER 2013, Indianapolis, IN, United States, 09/23/13. https://doi.org/10.1109/CLUSTER.2013.6702678

The scaling of many-task computing approaches in python on cluster supercomputers. / Lunacek, Monte; Braden, Jazcek; Hauser, Thomas.
2013 IEEE International Conference on Cluster Computing, CLUSTER 2013. 2013. 6702678 (Proceedings - IEEE International Conference on Cluster Computing, ICCC).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - The scaling of many-task computing approaches in python on cluster supercomputers

AU - Lunacek, Monte

AU - Braden, Jazcek

AU - Hauser, Thomas

PY - 2013

Y1 - 2013

N2 - We compare two packages for performing many-task computing (MTC) in Python: IPython Parallel and Celery. We describe these packages in detail and compare their features as applied to many-task computing on a cluster, including a scaling study using over 12,000 cores and several thousand tasks. We use mpi4py as a baseline for our comparisons. Our results suggest that Python is an excellent way to manage many-task computing and that no single technique is the obvious choice in every situation.

AB - We compare two packages for performing many-task computing (MTC) in Python: IPython Parallel and Celery. We describe these packages in detail and compare their features as applied to many-task computing on a cluster, including a scaling study using over 12,000 cores and several thousand tasks. We use mpi4py as a baseline for our comparisons. Our results suggest that Python is an excellent way to manage many-task computing and that no single technique is the obvious choice in every situation.

UR - https://www.scopus.com/pages/publications/84893610936

U2 - 10.1109/CLUSTER.2013.6702678

DO - 10.1109/CLUSTER.2013.6702678

M3 - Conference contribution

AN - SCOPUS:84893610936

SN - 9781479908981

T3 - Proceedings - IEEE International Conference on Cluster Computing, ICCC

BT - 2013 IEEE International Conference on Cluster Computing, CLUSTER 2013

T2 - 15th IEEE International Conference on Cluster Computing, CLUSTER 2013

Y2 - 23 September 2013 through 27 September 2013

ER -

The scaling of many-task computing approaches in python on cluster supercomputers

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this