Files in this item
Large-scale hierarchical k-means for heterogeneous many-core supercomputers
Item metadata
dc.contributor.author | Li, Lideng | |
dc.contributor.author | Yu, Teng | |
dc.contributor.author | Zhao, Wenlai | |
dc.contributor.author | Fu, Haohuan | |
dc.contributor.author | Wang, Chenyu | |
dc.contributor.author | Tan, Li | |
dc.contributor.author | Yang, Guangwen | |
dc.contributor.author | Thomson, John | |
dc.date.accessioned | 2018-11-13T13:30:05Z | |
dc.date.available | 2018-11-13T13:30:05Z | |
dc.date.issued | 2018-11-11 | |
dc.identifier.citation | Li , L , Yu , T , Zhao , W , Fu , H , Wang , C , Tan , L , Yang , G & Thomson , J 2018 , Large-scale hierarchical k-means for heterogeneous many-core supercomputers . in Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC '18) . IEEE Press , Piscataway , The International Conference for High Performance Computing, Networking, Storage, and Analysis , Dallas , Texas , United States , 11/11/18 . https://doi.org/10.5555/3291656.3291674 | en |
dc.identifier.citation | conference | en |
dc.identifier.isbn | 9781538683842 | |
dc.identifier.other | PURE: 255501866 | |
dc.identifier.other | PURE UUID: dd552c75-e08e-471f-9367-a8f909aa25fb | |
dc.identifier.other | Scopus: 85064131730 | |
dc.identifier.other | WOS: 000494258800013 | |
dc.identifier.uri | https://hdl.handle.net/10023/16441 | |
dc.description | Funding: J.Thomson and T.Yu are supported by the EPSRC grants ”Discovery” EP/P020631/1, ”ABC: Adaptive Brokerage for the Cloud” EP/R010528/1, and EU Horizon 2020 grant Team-Play: ”Time, Energy and security Analysis for Multi/Many-core heterogenous PLAtforms” (ICT-779882, https://teamplay- h2020.eu) | en |
dc.description.abstract | This paper presents a novel design and implementation of k-means clustering algorithm targeting the Sunway TaihuLight supercomputer. We introduce a multi-level parallel partition approach that not only partitions by dataflow and centroid, but also by dimension. Our multi-level (nkd) approach unlocks the potential of the hierarchical parallelism in the SW26010 heterogeneous many-core processor and the system architecture of the supercomputer. Our design is able to process large-scale clustering problems with up to 196,608 dimensions and over 160,000 targeting centroids, while maintaining high performance and high scalability, significantly improving the capability of k-means over previous approaches. The evaluation shows our implementation achieves performance of less than 18 seconds per iteration for a large-scale clustering case with 196,608 data dimensions and 2,000 centroids by applying 4,096 nodes (1,064,496 cores) in parallel, making k-means a more feasible solution for complex scenarios. | |
dc.format.extent | 11 | |
dc.language.iso | eng | |
dc.publisher | IEEE Press | |
dc.relation.ispartof | Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC '18) | en |
dc.rights | Copyright © 2018 IEEE Press. This work has been made available online in accordance with the publisher's policies. This is the author created accepted version manuscript following peer review and as such may differ slightly from the final published version. The final published version of this work is available at https://dl.acm.org/citation.cfm?id=3291674 | en |
dc.subject | Supercomputer | en |
dc.subject | Multi/many-core Processors | en |
dc.subject | Clustering | en |
dc.subject | Parallel computing | en |
dc.subject | QA75 Electronic computers. Computer science | en |
dc.subject | NDAS | en |
dc.subject | BDC | en |
dc.subject.lcc | QA75 | en |
dc.title | Large-scale hierarchical k-means for heterogeneous many-core supercomputers | en |
dc.type | Conference item | en |
dc.contributor.sponsor | EPSRC | en |
dc.contributor.sponsor | EPSRC | en |
dc.contributor.sponsor | European Commission | en |
dc.description.version | Postprint | en |
dc.contributor.institution | University of St Andrews. School of Computer Science | en |
dc.identifier.doi | https://doi.org/10.5555/3291656.3291674 | |
dc.date.embargoedUntil | 2018-11-11 | |
dc.identifier.url | https://dl.acm.org/citation.cfm?id=3291674 | en |
dc.identifier.grantnumber | EP/P020631/1 | en |
dc.identifier.grantnumber | EP/R010528/1 | en |
dc.identifier.grantnumber | 779882 | en |
This item appears in the following Collection(s)
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.