Investigating Performance Variability on Multi-core Processors

Abstract

Performance variability is important because it implies that performance is not always as good as it could have been. Running the same benchmark multiple times will give you different running times. A variable total runtime means that some runs are slower than others, and the slower runs must have been slowed down by something. As it is becoming increasingly difficult to improve sequential performance by building more complex CPUs, manufacturers have turned to multi-core CPUs to harvest performance of the increasing number of transistors. Therefore, parallel software is essential to fully utilize the performance potential of modern CPUs. Interestingly, the effect of performance variability is bigger for parallel software. Understanding and reducing performance variability in parallel software could thus lead to significant and important performance gains. This thesis will try to quantify performance variability in parallel software running on multi-core CPUs. It will also investigate in which extent dynamic frequency scaling affects performance variability. I ran the benchmark suite PARSEC on 3 computers and compared running times between different runs of the same benchmarks. With my setup, most benchmarks running with 4 threads vary in runtime between their fastest run and 5% slower. Some of them vary up to 10% slower. One exception had it's slowest run 28.6% slower than the fastest, for one of the configurations. Results also showed that disabling dynamic frequency scaling significantly reduced performance variability.