Understanding computation time : a critical discussion of time as a computational performance metric

Abstract

Computation time is an important performance metric that scientists and software engineers use to determine whether an algorithm is capable of running within a reasonable time frame. We provide an accessible critical review of the factors that influence computation time, highlighting problems in its reporting in current research and the negative practical impact that this has on developers, recommending best practice for its measurement and reporting. Discussing how computers and coders measure time, a discrepancy is exposed between best practice in the primarily theoretical field of computational complexity, and the difficulty for non-specialists in applying such theoretical findings. We therefore recommend establishing a better reporting practice, highlighting future work needed to expose the effects of poor reporting. Freely shareable templates are provided to help developers and researchers report this information more accurately, helping others to build upon their work, and thereby reducing the needless global duplication of computational and human effort.