Files in this item
Large-scale hierarchical k-means for heterogeneous many-core supercomputers
Item metadata
dc.contributor.author | Li, Lideng | |
dc.contributor.author | Yu, Teng | |
dc.contributor.author | Zhao, Wenlai | |
dc.contributor.author | Fu, Haohuan | |
dc.contributor.author | Wang, Chenyu | |
dc.contributor.author | Tan, Li | |
dc.contributor.author | Yang, Guangwen | |
dc.contributor.author | Thomson, John | |
dc.date.accessioned | 2018-11-13T13:30:05Z | |
dc.date.available | 2018-11-13T13:30:05Z | |
dc.date.issued | 2018-11-11 | |
dc.identifier | 255501866 | |
dc.identifier | dd552c75-e08e-471f-9367-a8f909aa25fb | |
dc.identifier | 85064131730 | |
dc.identifier | 000494258800013 | |
dc.identifier.citation | Li , L , Yu , T , Zhao , W , Fu , H , Wang , C , Tan , L , Yang , G & Thomson , J 2018 , Large-scale hierarchical k-means for heterogeneous many-core supercomputers . in Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC '18) . IEEE Press , Piscataway , The International Conference for High Performance Computing, Networking, Storage, and Analysis , Dallas , Texas , United States , 11/11/18 . https://doi.org/10.5555/3291656.3291674 | en |
dc.identifier.citation | conference | en |
dc.identifier.isbn | 9781538683842 | |
dc.identifier.uri | https://hdl.handle.net/10023/16441 | |
dc.description | Funding: J.Thomson and T.Yu are supported by the EPSRC grants ”Discovery” EP/P020631/1, ”ABC: Adaptive Brokerage for the Cloud” EP/R010528/1, and EU Horizon 2020 grant Team-Play: ”Time, Energy and security Analysis for Multi/Many-core heterogenous PLAtforms” (ICT-779882, https://teamplay- h2020.eu) | en |
dc.description.abstract | This paper presents a novel design and implementation of k-means clustering algorithm targeting the Sunway TaihuLight supercomputer. We introduce a multi-level parallel partition approach that not only partitions by dataflow and centroid, but also by dimension. Our multi-level (nkd) approach unlocks the potential of the hierarchical parallelism in the SW26010 heterogeneous many-core processor and the system architecture of the supercomputer. Our design is able to process large-scale clustering problems with up to 196,608 dimensions and over 160,000 targeting centroids, while maintaining high performance and high scalability, significantly improving the capability of k-means over previous approaches. The evaluation shows our implementation achieves performance of less than 18 seconds per iteration for a large-scale clustering case with 196,608 data dimensions and 2,000 centroids by applying 4,096 nodes (1,064,496 cores) in parallel, making k-means a more feasible solution for complex scenarios. | |
dc.format.extent | 11 | |
dc.format.extent | 926094 | |
dc.language.iso | eng | |
dc.publisher | IEEE Press | |
dc.relation.ispartof | Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC '18) | en |
dc.subject | Supercomputer | en |
dc.subject | Multi/many-core Processors | en |
dc.subject | Clustering | en |
dc.subject | Parallel computing | en |
dc.subject | QA75 Electronic computers. Computer science | en |
dc.subject | NDAS | en |
dc.subject | BDC | en |
dc.subject.lcc | QA75 | en |
dc.title | Large-scale hierarchical k-means for heterogeneous many-core supercomputers | en |
dc.type | Conference item | en |
dc.contributor.sponsor | EPSRC | en |
dc.contributor.sponsor | EPSRC | en |
dc.contributor.sponsor | European Commission | en |
dc.contributor.institution | University of St Andrews. School of Computer Science | en |
dc.identifier.doi | 10.5555/3291656.3291674 | |
dc.date.embargoedUntil | 2018-11-11 | |
dc.identifier.url | https://dl.acm.org/citation.cfm?id=3291674 | en |
dc.identifier.grantnumber | EP/P020631/1 | en |
dc.identifier.grantnumber | EP/R010528/1 | en |
dc.identifier.grantnumber | 779882 | en |
This item appears in the following Collection(s)
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.