I am trying to use Infection (https://infection.github.io/) on a Laravel 10 codebase and am running into issues when running Infection with more than 1 thread. Running it single threaded is not ideal as it takes nearly 20 minutes to complete.
Infection does run with multiple threads, but it reports a much higher MSI score then single threaded. My own debugging of these issues point me to the database being used in feature tests as being the problem, specifically the Infection threads seem to be using only a single database and the automated database clean up and creation (which work fine with artisan test with multiple threads) seem to cause basically all feature tests to run into database related exceptions. For example a table that already exists when it doesn't expect it yet or a table that does not exist when it should. This causes Infection to mark a mutation as killed, but wasn't the test legitimately catching the issue, just threads fighting over the database.
Unfortunately, I have only started developing in Laravel recently and have not found a good way to ensure that in the case of Infection running the tests it will properly use separate databases. I have tried to somehow introduce Infection's test_token into the database name used during testing, but could not figure out a good way to do this.
I have tried overriding the database creation in laravel via the exposed Exposed environment variables TEST_TOKEN=<int> that is provided by Infection. But I haven't succeeded.
One thing I have tried is adding this to the setup of tests:
if (getenv('INFECTION_RUNNING') === 'true') {
if (!isset(self::$uniqueToken)) {
self::$uniqueToken = uniqid('thread_', true) . '_' . random_int(1000, 9999);
}
// Dynamically set the database for each test thread during Infection run
$databaseName = 'laravel_test_' . self::$uniqueToken; // Unique database for each Infection thread
// Set database dynamically for Infection
config()->set('database.connections.mysql.database', $databaseName);
// Create the database if it doesn't exist (specific for Infection run)
\DB::statement("CREATE DATABASE IF NOT EXISTS `$databaseName`");
// Run migrations for the Infection test database
\Artisan::call('migrate');
}
Another thing I tried is adding a artisan command that creates the database
/**
* Command to run mutation tests via the infection library.
*
* This command is a modified version of the `TestCommand` class from the Laravel framework.
*/
class TestMutation extends Command
{
/**
* Execute the console command.
*
* @return mixed
*/
public function handle()
{
$this->clearEnv();
$parallel = $this->option('parallel');
$threads = 4;
foreach (range(1, $threads) as $thread) {
$this->createDatabase();
}
$process = (new Process(
array_merge(
// Binary ...
$this->binary(),
// Arguments ...
$this->getInfectionArguments()
),
null,
// Envs ...
$this->getInfectionEnvironmentVariables(),
))->setTimeout(null);
$exitCode = 1;
try {
$exitCode = $process->run(function ($type, $line) {
$this->output->write($line);
});
} catch (ProcessSignaledException $e) {
if (extension_loaded('pcntl') && $e->getSignal() !== SIGINT) {
throw $e;
}
}
if ($exitCode === 0 && $this->option('coverage')) {
if (!$this->usingPest() && $this->option('parallel')) {
$this->newLine();
}
$coverage = Coverage::report($this->output);
$exitCode = (int)($coverage < $this->option('min'));
if ($exitCode === 1) {
$this->output->writeln(sprintf(
"\n <fg=white;bg=red;options=bold> FAIL </> Code coverage below expected:<fg=red;options=bold> %s %%</>. Minimum:<fg=white;options=bold> %s %%</>.",
number_format($coverage, 1),
number_format((float)$this->option('min'), 1)
));
}
}
return $exitCode;
}
/**
* Get the PHP binary to execute.
*
* @return array
*/
protected function binary()
{
$command = ['vendor/bin/infection'];
if ('phpdbg' === PHP_SAPI) {
return array_merge([PHP_BINARY, '-qrr'], $command);
}
return array_merge([PHP_BINARY], $command);
}
protected function getInfectionArguments(): array
{
return [
'--configuration=/usr/src/.ci-config/.infection/infection.json5',
'--threads=2',
'--coverage=coverage',
];
}
/**
* Get the array of environment variables for running Paratest.
*
* @return array
*/
protected function getInfectionEnvironmentVariables(): array
{
return [];
}
/**
* Clears any set Environment variables set by Laravel if the --env option is empty.
*
* @return void
*/
protected function clearEnv(): void
{
if (!$this->option('env')) {
$vars = self::getEnvironmentVariables(
$this->laravel->environmentPath(),
$this->laravel->environmentFile()
);
$repository = Env::getRepository();
foreach ($vars as $name) {
$repository->clear($name);
}
}
}
/**
* @param string $path
* @param string $file
* @return array
*/
protected static function getEnvironmentVariables(string $path, string $file): array
{
try {
$content = StoreBuilder::createWithNoNames()
->addPath($path)
->addName($file)
->make()
->read();
} catch (InvalidPathException $e) {
return [];
}
$vars = [];
foreach ((new Parser())->parse($content) as $entry) {
$vars[] = $entry->getName();
}
return $vars;
}
/**
* @return void
*/
private function createDatabase(): void
{
Config::set('database.connections.mysql.database', null);
DB::purge('mysql');
$database_name = env('DB_DATABASE') . '_' . ParallelTesting::token();
echo $database_name;
$password = 'laravelpw';
// Artisan::command("db GRANT ALL PRIVILEGES ON $database_name .* TO laravel@'%' IDENTIFIED BY $password;", function () {
// echo 'GRANT ALL PRIVILEGES ON $database_name .* TO laravel@\'%\' IDENTIFIED BY $password;';
// });
//
//
// // Create database via PDO
// app('db')->getPdo()->exec('CREATE DATABASE IF NOT EXISTS `' . $database_name . '`');
DB::connection()->statement('GRANT ALL PRIVILEGES ON hello.* TO \'laravel\'@\'%\' IDENTIFIED BY \'password\';');
DB::connection()->statement('grant all privileges on *.* to \'<user>\'@\'localhost\';');
DB::connection()->statement('CREATE DATABASE hello.*');
// Reconnect the database for when this is called by Artisan::call()
Config::set('database.connections.mysql.database', $database_name);
DB::reconnect();
}
}
Another thing I tried was using a ParallelTestServiceProvider:
class ParallelTestsServiceProvider extends ServiceProvider
{
/**
* Bootstrap services.
*/
public function boot(): void
{
ParallelTesting::resolveTokenUsing(static function () {
echo 'doing something' . PHP_EOL;
$base_token = 'TEST_' . getmypid();
$infection_token = getenv('TEST_TOKEN');
$infection_token = is_string($infection_token) ? '-' . $infection_token : '';
return $base_token . $infection_token;
});
ParallelTesting::setUpProcess(static function () {
echo 'doing setting up process' . PHP_EOL;
Config::set('database.connections.mysql.database', null);
DB::purge('mysql');
$database_name = env('DB_DATABASE') . '_' . ParallelTesting::token();
echo $database_name;
// Create database via PDO
app('db')->getPdo()->exec('CREATE DATABASE IF NOT EXISTS `' . $database_name . '`');
// Reconnect the database for when this is called by Artisan::call()
Config::set('database.connections.mysql.database', $database_name);
DB::reconnect();
});
}
}
We use a Mysql database in the project. And laravel 10.
I also can't find other examples of people doing this online.
Any help in solving the above issue or a different way in which to get multi-threaded Infection working would be appreciated.
