Software Implementation: Non-parametric Hierarchical Performance Testing (HPT)


  • Tianshi Chen, Qi Guo, Olivier Temam, Yue Wu, Yungang Bao, Zhiwei Xu, and Yunji Chen, "Statistical Performance Comparisons of Computers," IEEE Transactions on Computers, in press.

  • Tianshi Chen, Yunji Chen, Qi Guo, Olivier Temam, Yue Wu, and Weiwu Hu,  "Statistical Performance Comparisons of Computers," In Proceedings of 18th International Symposium on High-Performance Computer Architecture (HPCA-18), New Orleans, Louisiana, 2012. [Slides]

  • Motivation: As a fundamental task in computer architecture research, performance comparison has been continuously hampered by the variability of computer performance. In traditional performance comparisons, the impact of performance variability is usually ignored (i.e., the means of performance observations are compared regardless of the variability), or in the few cases directly addressed with t-statistics without checking the number and normality of performance observations. In this paper, we formulate a performance comparison as a statistical task, and empirically illustrate why and how common practices can lead to incorrect comparisons. We suggest a non-parametric Hierarchical Performance Testing (HPT) framework for performance comparison, which is significantly more practical than standard t-statistics because it does not require to collect a large number of performance observations in order to achieve a normal distribution of sample mean. In particular, the suggested HPT can facilitate quantitative performance comparison, in which the performance speedup of one computer over anotheris statistically evaluated. This HPT framework has been implemented as an open-source software, and integrated in the PARSEC 3.0 benchmark suite.

    Descriptions: The Hierarchical Performance Testing (HPT) is a statistical technique dedicated to comparing the performance of different computers. The HPT adopts Wilcoxon rank-sum test and Wilcoxon Signed-rank test to conduct pair-wise performance comparison, thus does not rely on normally distributed mean performance of each computer.

    Download HPT for MS Windows OS, or PARSEC 3.0 (with HPT).

    Maintained by Tianshi Chen