IPPD: Integrated End-to-End Performance Prediction and Diagnosis for Extreme Scientific Workflows

Scientific workflows are powerful tools for the management of scalable experiments, often composed of complex tasks running on distributed resources. Existing cyberinfrastructure provides components that can be utilized within repeatable workflows. However, data and computing advances continuously change the way scientific workflows get developed and executed, pushing the scientific activity to be more data-driven, heterogeneous, and collaborative.

This report details the accomplishments from the ASCR funded project “Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows” under the award numbers FWP-66406 and DE-SC0012630, with a focus on the UC San Diego (Award No. DE-SC0012630) part of the accomplishments. We refer to the project as IPPD.

Scientific workflows execute on a loosely connected set of distributed and heterogeneous computational resources. The Integrated End-to-End Performance Prediction and Diagnosis (IPPD) project contributes to clear understanding of the factors that influence the performance and potential optimization of scientific workflows. IPPD addressed three core issues in order to provide insights into workflow execution that can be used to both explain and optimize their execution: