Continuous integration and delivery for HPC: Using Singularity and Jenkins

Zebula Sampedro; Aaron Holt; Thomas Hauser

doi:10.1145/3219104.3219147

Continuous integration and delivery for HPC: Using Singularity and Jenkins

Zebula Sampedro, Aaron Holt, Thomas Hauser

University of Colorado Boulder

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

12 Scopus citations

Abstract

Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer [17] to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.

Original language	English
Title of host publication	Practice and Experience in Advanced Research Computing 2018
Subtitle of host publication	Seamless Creativity, PEARC 2018
Publisher	Association for Computing Machinery
ISBN (Print)	9781450364461
DOIs	https://doi.org/10.1145/3219104.3219147
State	Published - Jul 22 2018
Event	2018 Practice and Experience in Advanced Research Computing Conference: Seamless Creativity, PEARC 2018 - Pittsburgh, United States Duration: Jul 22 2017 → Jul 26 2017

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	2018 Practice and Experience in Advanced Research Computing Conference: Seamless Creativity, PEARC 2018
Country/Territory	United States
City	Pittsburgh
Period	07/22/17 → 07/26/17

Keywords

Containers
Continuous delivery
Continuous deployment
Continuous integration
MFIX-Exa
Singularity
Software automation
Software builds

Access to Document

10.1145/3219104.3219147

Cite this

@inproceedings{d02bde51068e4409900c2390901fb1fb,

title = "Continuous integration and delivery for HPC: Using Singularity and Jenkins",

abstract = "Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer [17] to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.",

keywords = "Containers, Continuous delivery, Continuous deployment, Continuous integration, MFIX-Exa, Singularity, Software automation, Software builds",

author = "Zebula Sampedro and Aaron Holt and Thomas Hauser",

note = "Publisher Copyright: {\textcopyright} 2018 Copyright held by the owner/author(s).; 2018 Practice and Experience in Advanced Research Computing Conference: Seamless Creativity, PEARC 2018 ; Conference date: 22-07-2017 Through 26-07-2017",

year = "2018",

month = jul,

day = "22",

doi = "10.1145/3219104.3219147",

language = "English",

isbn = "9781450364461",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

booktitle = "Practice and Experience in Advanced Research Computing 2018",

address = "United States",

}

Sampedro, Z, Holt, A & Hauser, T 2018, Continuous integration and delivery for HPC: Using Singularity and Jenkins. in Practice and Experience in Advanced Research Computing 2018: Seamless Creativity, PEARC 2018., a6, ACM International Conference Proceeding Series, Association for Computing Machinery, 2018 Practice and Experience in Advanced Research Computing Conference: Seamless Creativity, PEARC 2018, Pittsburgh, United States, 07/22/17. https://doi.org/10.1145/3219104.3219147

Continuous integration and delivery for HPC: Using Singularity and Jenkins. / Sampedro, Zebula; Holt, Aaron; Hauser, Thomas.
Practice and Experience in Advanced Research Computing 2018: Seamless Creativity, PEARC 2018. Association for Computing Machinery, 2018. a6 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Continuous integration and delivery for HPC

T2 - 2018 Practice and Experience in Advanced Research Computing Conference: Seamless Creativity, PEARC 2018

AU - Sampedro, Zebula

AU - Holt, Aaron

AU - Hauser, Thomas

PY - 2018/7/22

Y1 - 2018/7/22

N2 - Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer [17] to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.

AB - Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer [17] to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.

KW - Containers

KW - Continuous delivery

KW - Continuous deployment

KW - Continuous integration

KW - MFIX-Exa

KW - Singularity

KW - Software automation

KW - Software builds

UR - https://www.scopus.com/pages/publications/85051424130

U2 - 10.1145/3219104.3219147

DO - 10.1145/3219104.3219147

M3 - Conference contribution

AN - SCOPUS:85051424130

SN - 9781450364461

T3 - ACM International Conference Proceeding Series

BT - Practice and Experience in Advanced Research Computing 2018

PB - Association for Computing Machinery

Y2 - 22 July 2017 through 26 July 2017

ER -

Continuous integration and delivery for HPC: Using Singularity and Jenkins

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this