Automatic Runtime Scheduling Via Directed Acyclic Graphs for CFD Applications

The order of execution of computational kernels for Single-Program, Multiple-Data (SPMD) programs is usually determined at compile time. These static predetermined schedules can lead to performance issues at runtime, and are difficult to implement for inhomogeneous situations, such as variable-order or multi-physics applications. It is especially challenging to generate performant schedules when it is unknown whether specific kernels require execution, as a function of user inputs, or the kernel execution time changes dependent on the hardware. This paper presents a solution to this problem by dynamically scheduling computational kernels at runtime using directed acyclic graphs to track the data dependencies between kernels. This system is specifically designed to leverage existing computational infrastructure as much as possible, facilitating the extension to legacy applications. This scheduling system is demonstrated using the eddy high-order multi-physics solver developed at NASA. The details regarding the implementation, our experiences using this system, and performance are discussed.