Application of Deep Learning on Integrating Prediction, Provenance, and Optimization
In this research, we investigated two approaches to detect job anomalies and/or contention for large scale computing efforts:
- Preemptive job scheduling using binomial classification long short-term memory networks
- Forecasting intra-node computing loads from the active jobs and additional job(s)
For approach 1, we achieved a 14% improvement in computational resources utilization and an overall classification accuracy of 85% on real tasks executed in a High Energy Physics computing workflow. For this paper, we present the preliminary results used in second approach.
Link