Skip to main content
1 of 5
Doc Brown
  • 220.3k
  • 35
  • 410
  • 623

are test suites typically run on a installed version of the software or on the source code its self?

This is not a question of "trend" or popularity. It is a question of practical matters, of the trade-off between "ease-of-work" vs. the risk of getting "false positive or false negative reports" from the test suite.

On one hand, the cycle of "testing / debugging / changing code / rerun the tests again" should be as quick, as short and painless as possible. That's best achieved when your tests can be executed directly for a debuggable version of your code, and when the test suit runs only a few seconds. When a test fails, it is great when you can immediately let the IDE stop execution and show the code including its stack trace in the debugger.

On the other hand, the process of compiling / packaging a program into a deployable component itself, or altering the program's runtime environment may introduce errors which are not showing up in the unpackaged debug version of your program. Hence, having tests in place for that case is also a good idea.

As with any trade-off, there is no totally simple recipe for dealing with this situation. It depends heavily on the kind of software you are developing, how different the development and production environments are, how long the packaging process takes, how quick the test suit runs and how cumbersome it is to debug a packaged version in comparison to an unpackaged version. As a general strategy, you may consider to split your tests:

  • Tests which you run in the debug environment, and where you are quite confident that if they don't fail there, they won't fail in the production-like environment as well. Unit tests fall typically under this category, but not necessarily exclusively, you can have also integration tests, end-to-end tests, categorization tests there, as long as they don't take more than a minute to run.

  • tests which you run only in the production-like test environment (hence on a packaged version). If you are using a CI (continuous integration) process, you may place these tests on the CI server, and let the packaging and execution of the test suit happen there (for example, as part of a nightly or hourly build, whatever you do in your enviroment). There it will not really matter if the test suite requires an hour to complete. But even if you do not use a CI server, you can run these tests, for example, shortly before you create a new release of your library (but not during normal, daily development),

Still, it is a good idea to design all tests, including the tests for the production-like environment in a way you can run them in your development & debug environment as well, since it makes it a lot easier to find the root cause of a failing test. That does not mean you will run them always there, but, for example, in case the CI server has reported a failure during the last nightly build.

Doc Brown
  • 220.3k
  • 35
  • 410
  • 623