Files in this item
A framework for generating informative benchmark instances
Item metadata
dc.contributor.author | Dang, Nguyen | |
dc.contributor.author | Akgun, Ozgur | |
dc.contributor.author | Espasa Arxer, Joan | |
dc.contributor.author | Miguel, Ian James | |
dc.contributor.author | Nightingale, Peter | |
dc.contributor.editor | Solon, Christine | |
dc.date.accessioned | 2022-07-28T16:30:04Z | |
dc.date.available | 2022-07-28T16:30:04Z | |
dc.date.issued | 2022-07-23 | |
dc.identifier | 279532119 | |
dc.identifier | c341e68d-f4c1-4ab5-b77b-0693aa1cc90f | |
dc.identifier | 85135706603 | |
dc.identifier.citation | Dang , N , Akgun , O , Espasa Arxer , J , Miguel , I J & Nightingale , P 2022 , A framework for generating informative benchmark instances . in C Solon (ed.) , 28th International Conference on Principles and Practice of Constraint Programming (CP 2022) . , 18 , Leibniz International Proceedings in Informatics (LIPIcs) , vol. 235 , Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing , Dagstuhl . https://doi.org/10.4230/LIPIcs.CP.2022.18 | en |
dc.identifier.isbn | 9783959772402 | |
dc.identifier.issn | 1868-8969 | |
dc.identifier.other | ORCID: /0000-0002-6930-2686/work/116597866 | |
dc.identifier.other | ORCID: /0000-0001-9519-938X/work/116598018 | |
dc.identifier.other | ORCID: /0000-0002-2693-6953/work/116598356 | |
dc.identifier.uri | https://hdl.handle.net/10023/25744 | |
dc.description | Funding: Nguyen Dang: is a Leverhulme Early Career Fellow; Ian Miguel: supported by EPSRC EP/V027182/1. | en |
dc.description.abstract | Benchmarking is an important tool for assessing the relative performance of alternative solving approaches. However, the utility of benchmarking is limited by the quantity and quality of the available problem instances. Modern constraint programming languages typically allow the specification of a class-level model that is parameterised over instance data. This separation presents an opportunity for automated approaches to generate instance data that define instances that are graded (solvable at a certain difficulty level for a solver) or can discriminate between two solving approaches. In this paper, we introduce a framework that combines these two properties to generate a large number of benchmark instances, purposely generated for effective and informative benchmarking. We use five problems that were used in the MiniZinc competition to demonstrate the usage of our framework. In addition to producing a ranking among solvers, our framework gives a broader understanding of the behaviour of each solver for the whole instance space; for example by finding subsets of instances where the solver performance significantly varies from its average performance. | |
dc.format.extent | 18 | |
dc.format.extent | 1119910 | |
dc.language.iso | eng | |
dc.publisher | Schloss Dagstuhl- Leibniz-Zentrum fur Informatik GmbH, Dagstuhl Publishing | |
dc.relation.ispartof | 28th International Conference on Principles and Practice of Constraint Programming (CP 2022) | en |
dc.relation.ispartofseries | Leibniz International Proceedings in Informatics (LIPIcs) | en |
dc.subject | Instance generation | en |
dc.subject | Benchmarking | en |
dc.subject | Constraint programming | en |
dc.subject | QA75 Electronic computers. Computer science | en |
dc.subject | QA76 Computer software | en |
dc.subject | DAS | en |
dc.subject.lcc | QA75 | en |
dc.subject.lcc | QA76 | en |
dc.title | A framework for generating informative benchmark instances | en |
dc.type | Conference item | en |
dc.contributor.sponsor | The Leverhulme Trust | en |
dc.contributor.sponsor | EPSRC | en |
dc.contributor.institution | University of St Andrews. Centre for Interdisciplinary Research in Computational Algebra | en |
dc.contributor.institution | University of St Andrews. School of Computer Science | en |
dc.contributor.institution | University of St Andrews. Sir James Mackenzie Institute for Early Diagnosis | en |
dc.identifier.doi | 10.4230/LIPIcs.CP.2022.18 | |
dc.identifier.grantnumber | ECF-2020-168 | en |
dc.identifier.grantnumber | EP/V027182/1 | en |
This item appears in the following Collection(s)
Items in the St Andrews Research Repository are protected by copyright, with all rights reserved, unless otherwise indicated.