Skip to content

Test runner's --test-rerun-failures could swallow actual failures on retries #63424

@atlowChemi

Description

@atlowChemi

Version

26.1.0

Platform

Darwin ARM64

Subsystem

test_runner

What steps will reproduce the bug?

The following example test file will reproduce the bug:

// Demonstrates that `node --test --test-rerun-failures` can mark a failing test as passing on retry without executing its body.
// Root cause: the runtime disambiguator at lib/internal/test_runner/test.js:793-801 keys tests purely by `file:line:column`.
// A `t.test()` registered inside a factory function gets the same source location regardless of which parent invoked the factory.
// The disambiguator counter is assigned in execution order across the whole run, so on retry - when the set of running tests changes,
// the surviving failing test inherits a counter slot that, in attempt 0, belonged to a different (passing) sibling.
// Node matches by that slot, replaces `this.fn` with a synthetic noop replay, and reports the failure as a pass.

import { describe, it } from 'node:test';
import { strict as assert } from 'node:assert';

function makeSuite(shouldPass, label) {
  return async (t) => {
    await t.test('inner', async () => {
      if (!shouldPass) assert.fail(`${label} should fail`);
    });
  };
}

describe('parents', { concurrency: false }, () => {
  it('A passes', makeSuite(true,  'A'));
  it('B fails',  makeSuite(false, 'B'));   // the only real failure
  it('C passes', makeSuite(true,  'C'));
});

Executed with:

node --test --test-rerun-failures=rerun.json repro.mjs   # attempt 0
# B fails as expected; rerun.json now contains A and C plus their
# inner subtests keyed as "repro.mjs:13:13" and "repro.mjs:13:13:(1)"

node --test --test-rerun-failures=rerun.json repro.mjs   # retry
# all 6 tests reported as passing, exit code 0
# B's assert.fail was never executed

How often does it reproduce? Is there a required condition?

Happens consistently

What is the expected behavior? Why is that the expected behavior?

The test inner should not be replaced with a noop for B fails, allowing it to actually execute

What do you see instead?

The inner test of B fails is skipped completely, due to a erroneous disambiguation

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    confirmed-bugIssues with confirmed bugs.test_runnerIssues and PRs related to the test runner subsystem.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions