Files in this item
Optimization towards efficiency and stateful of dispel4py
Item metadata
dc.contributor.author | Liang, Liang | |
dc.contributor.author | Zhang, Heting | |
dc.contributor.author | Yang, Guang | |
dc.contributor.author | Heinis, Thomas | |
dc.contributor.author | Filgueira, Rosa | |
dc.date.accessioned | 2024-06-14T12:30:10Z | |
dc.date.available | 2024-06-14T12:30:10Z | |
dc.date.issued | 2023-11-01 | |
dc.identifier | 293775834 | |
dc.identifier | d0b920fc-0a43-407e-afef-63ff3db022a3 | |
dc.identifier | 85178112986 | |
dc.identifier.citation | Liang , L , Zhang , H , Yang , G , Heinis , T & Filgueira , R 2023 , Optimization towards efficiency and stateful of dispel4py . in Proceedings of the SC '23 workshops of the international conference on high performance computing, network, storage, and analysis (SC-W '23) : Nov 12-17, 2023 | Denver, CO . ACM , pp. 2021–2032 , 18th Workshop on Workflows in Support of Large-Scale Science (WORKS 2023) , Denver , Colorado , United States , 12/11/23 . https://doi.org/10.1145/3624062.3624281 | en |
dc.identifier.citation | conference | en |
dc.identifier.isbn | 9798400707858 | |
dc.identifier.uri | https://hdl.handle.net/10023/30023 | |
dc.description.abstract | Scientific workflows bridge scientific challenges with computational resources. While dispel4py, a stream-based workflow system, offers mappings to parallel enactment engines like MPI or Multiprocessing, its optimization primarily focuses on dynamic process-to-task allocation for improved performance. An efficiency gap persists, particularly with the growing emphasis on conserving computing resources. Moreover, the existing dynamic optimization lacks support for stateful applications and grouping operations. To address these issues, our work introduces a novel hybrid approach for handling stateful operations and groupings within workflows, leveraging a new Redis mapping. We also propose an auto-scaling mechanism integrated into dispel4py’s dynamic optimization. Our experiments showcase the effectiveness of auto-scaling optimization, achieving efficiency while upholding performance. In the best case, auto-scaling reduces dispel4py’s runtime to 87% compared to the baseline, using only 76% of process resources. Importantly, our optimized stateful dispel4py demonstrates a remarkable speedup, utilizing just 32% of the runtime compared to the contender. To address these issues, our work introduces a novel hybrid approach for handling stateful operations and groupings within workflows, leveraging a new Redis mapping. We also propose an auto-scaling mechanism integrated into dispel4py’s dynamic optimization. Our experiments showcase the effectiveness of autoscaling optimization, achieving efficiency while upholding performance. In the best case, auto-scaling reduces dispel4py’s runtime to 87% compared to the baseline, using only 76% of process resources. Importantly, our optimized stateful dispel4py demonstrates a remarkable speedup, utilizing just 32% of the runtime compared to the contender. | |
dc.format.extent | 1430728 | |
dc.language.iso | eng | |
dc.publisher | ACM | |
dc.relation.ispartof | Proceedings of the SC '23 workshops of the international conference on high performance computing, network, storage, and analysis (SC-W '23) | en |
dc.subject | Scientific workflows | en |
dc.subject | Stream-based workflow | en |
dc.subject | Workflow optimization | en |
dc.subject | Auto-scaling | en |
dc.subject | Stateful application | en |
dc.subject | dispel4py | en |
dc.subject | QA75 Electronic computers. Computer science | en |
dc.subject | QA76 Computer software | en |
dc.subject | NS | en |
dc.subject.lcc | QA75 | en |
dc.subject.lcc | QA76 | en |
dc.title | Optimization towards efficiency and stateful of dispel4py | en |
dc.type | Conference item | en |
dc.contributor.institution | University of St Andrews. School of Computer Science | en |
dc.identifier.doi | 10.1145/3624062.3624281 |
This item appears in the following Collection(s)
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.