Files in this item
Scalable adaptive optimizations for stream-based workflows in multi-HPC-clusters and cloud infrastructures
Item metadata
dc.contributor.author | Liang, Liang | |
dc.contributor.author | Filgueira, Rosa | |
dc.contributor.author | Yan, Yan | |
dc.contributor.author | Heinis, Thomas | |
dc.date.accessioned | 2022-10-07T23:39:16Z | |
dc.date.available | 2022-10-07T23:39:16Z | |
dc.date.issued | 2022-03 | |
dc.identifier | 277807121 | |
dc.identifier | 2140963d-057b-441c-b7a2-3f48e5b5dd31 | |
dc.identifier | 85118733552 | |
dc.identifier.citation | Liang , L , Filgueira , R , Yan , Y & Heinis , T 2022 , ' Scalable adaptive optimizations for stream-based workflows in multi-HPC-clusters and cloud infrastructures ' , Future Generation Computer Systems , vol. 128 , pp. 102-116 . https://doi.org/10.1016/j.future.2021.09.036 | en |
dc.identifier.issn | 0167-739X | |
dc.identifier.other | Bibtex: 73b0b90895dc48fea5d91646e69bc2d8 | |
dc.identifier.uri | https://hdl.handle.net/10023/26161 | |
dc.description | Funding: This work is partially supported by the EU H2020 project DARE, No. 777413; and by Google Cloud Platform research credits program. | en |
dc.description.abstract | This work presents three new adaptive optimization techniques to maximize the performance of dispel4py workflows. dispel4py is a parallel Python-based stream-oriented dataflow framework that acts as a bridge to existing parallel programming frameworks like MPI or Python multiprocessing. When a user runs a dispel4py workflow, the original framework performs a fixed workload distribution among the processes available for the run. This allocation does not take into account the features of the workflows, which can cause scalability issues, especially for data-intensive scientific workflows. Our aim, therefore, is to improve the performance of dispel4py workflows by testing different workload strategies that automatically adapt to workflows at runtime. For achieving this objective, we have implemented three new techniques, called Naive Assignment, Staging and Dynamic Scheduling. We have evaluated our proposal with several workflows from different domains and across different computing resources. The results show that our proposed techniques have significantly (up to 10X) improved the performance of the original dispel4py framework. | |
dc.format.extent | 15 | |
dc.format.extent | 3059433 | |
dc.language.iso | eng | |
dc.relation.ispartof | Future Generation Computer Systems | en |
dc.subject | Scientific workflow | en |
dc.subject | Stream-based workflow | en |
dc.subject | Workflow optimization | en |
dc.subject | dispel4py | en |
dc.subject | Distributed systems | en |
dc.subject | QA75 Electronic computers. Computer science | en |
dc.subject | QA76 Computer software | en |
dc.subject | I | en |
dc.subject | AC | en |
dc.subject.lcc | QA75 | en |
dc.subject.lcc | QA76 | en |
dc.title | Scalable adaptive optimizations for stream-based workflows in multi-HPC-clusters and cloud infrastructures | en |
dc.type | Journal article | en |
dc.contributor.institution | University of St Andrews. School of Computer Science | en |
dc.identifier.doi | https://doi.org/10.1016/j.future.2021.09.036 | |
dc.description.status | Peer reviewed | en |
dc.date.embargoedUntil | 2022-10-08 | |
dc.identifier.url | https://researchportal.hw.ac.uk/en/publications/scalable-adaptive-optimizations-for-stream-based-workflows-in-mul | en |
This item appears in the following Collection(s)
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.