One idea with test-driven design is to run all your unit tests when building a project, which allows you to catch functional regressions quickly, before committing the changes that caused them. For this to work, though, all your tests need to complete quickly.

If your benchmarks all run quickly, they can be included in this framework. Also, the whole point of doing so would be to catch major performance regressions quickly -- so, for this usage, you want to flag taking too long as an error.

One idea with test-driven design is to run all your unit tests when building a project, which allows you to catch functional regressions quickly, before committing the changes that caused them. For this to work, though, all your tests need to complete quickly.

If your benchmarks all run quickly, they can be included in your test framework (I don't know if that is what Caliper is supposed to be for, but Google is heavily into TDD...). In any case, the whole point of adding benchmarks to your test-driven infrastructure would be to catch major performance regressions quickly -- so, for that kind of usage, you'd want to flag taking too long as an error.