how to compare benchmarks in perl

Question

using the standard Benchmark module, I start to write some code like as usual, however I have one doubt: How can I compare two different versions of the same module?

For instance, I have a stable module version 1.0 and some parts may need some attention. So I decided to start write the version 1.1 in another branch and...

I can write a perl benchmark script that can output a series of timethese calls to compare different aspects of the code and run against both versions then check the output.

My question is... how can I compare two benchmark outputs? is there a benchmark diff tool?

I was checking around metacpan and find nothing to help me. And what google gives me is examples of benchmark using cmpthese but it compares two codes in the same codebase.

My other option is to create a Package::Under::Test2 that is a copy of the original package and perform modifications and in the end I just substitute rename the package but it seems... ugly

For instance, on golang we have the tool benchstat that can compare two benchmarks outputs and produce some statistics.

suggestions?

Answer 1

Run the benchmark twice with different settings for @INC. One run has the path for the old Perl module, and one run has the path for the new perl module.

$ perl -I/path/to/old benchmark.pl
$ perl -I/path/to/new benchmark.pl

You don't care so much about the percentages because you can look at the iterations per second.

But, unless it's a radically different implementation of the module, I wouldn't expect any significant differences between two minor versions of the same module. Remember that any difference of about 5% from Benchmark.pm is just benchmarking nois; that's why Dumbbench exists (and you should read Steffan's various posts about measurement theory).

Answer 2

Rather than benchmarking, how about profiling? You know the changes you're making, so running the two versions through Devel::NYTProf will show you in colour what effect each line is having. It will help direct where you want to look for performance gains.

It's a fun module and easy to get started. Results are displayed in Web 1.0 html. I have found it a little addictive, so set yourself a halting condition to avoid chasing diminishing returns and adding unnecessary complexity.