UT unstable test tiflashDDLTestSuite.TestTiFlashReplicaPartitionTableNormal

[tiancaiamao]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Found in CI https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_unit_test/detail/tidb_ghpr_unit_test/35082/pipeline/

2. What did you expect to see? (Required)

3. What did you see instead (Required)

It's werid... the test itself run success:

[2022-02-10T10:19:34.679Z] PASS: ddl_tiflash_test.go:270: tiflashDDLTestSuite.TestTiFlashReplicaPartitionTableNormal	7.175s

Then when the test finish and cleanup, the process hang, and then timeout:'

[2022-02-10T10:19:34.680Z] 2022-02-10 18:17:46.303891 I | httptest.Server blocked in Close after 5 seconds, waiting for connections:
[2022-02-10T10:19:34.680Z]   *net.TCPConn 0xc005bb4860 127.0.0.1:46380 in state active
[2022-02-10T10:19:34.680Z] coverage: 19.0% of statements
[2022-02-10T10:19:34.680Z] panic: test timed out after 2m0s

There are some error message like this:

[2022-02-10T10:19:34.679Z] [2022/02/10 18:17:41.303 +08:00] [ERROR] [ddl_tiflash_api.go:229] ["Update TiFlash status address failed"] [error="Get \"[http://127.0.0.1:44273/config\](http://127.0.0.1:44273/config/)": dial tcp 127.0.0.1:44273: connect: connection refused"] [errorVerbose="Get \"[http://127.0.0.1:44273/config\](http://127.0.0.1:44273/config/)": dial tcp 127.0.0.1:44273: connect: connection refused\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/juju_adaptor.go:15\ngithub.com/pingcap/tidb/ddl.getTiflashHTTPAddr\n\t/home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_tiflash_api.go:93\ngithub.com/pingcap/tidb/ddl.(*ddl).UpdateTiFlashHTTPAddress\n\t/home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_tiflash_api.go:154\ngithub.com/pingcap/tidb/ddl.(*ddl).pollTiFlashReplicaStatus\n\t/home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_tiflash_api.go:228\ngithub.com/pingcap/tidb/ddl.(*ddl).PollTiFlashRoutine\n\t/home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_tiflash_api.go:433\ngithub.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1\n\t/home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/util/wait_group_wrapper.go:33\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"]

From the goroutine stack, Close operation blocked on a waitgroup:

[2022-02-10T10:19:34.685Z] goroutine 750 [semacquire]:
[2022-02-10T10:19:34.685Z] sync.runtime_Semacquire(0xc00062de78)
[2022-02-10T10:19:34.685Z] 	/usr/local/go/src/runtime/sema.go:56 +0x45
[2022-02-10T10:19:34.685Z] sync.(*WaitGroup).Wait(0xc00062de70)
[2022-02-10T10:19:34.685Z] 	/usr/local/go/src/sync/waitgroup.go:130 +0x65
[2022-02-10T10:19:34.685Z] net/http/httptest.(*Server).Close(0xc00062de30)
[2022-02-10T10:19:34.685Z] 	/usr/local/go/src/net/http/httptest/server.go:249 +0x232
[2022-02-10T10:19:34.685Z] github.com/pingcap/tidb/ddl_test.(*tiflashDDLTestSuite).TearDownSuite(0xc000dbe9c0, 0xc0062d73b0)
[2022-02-10T10:19:34.685Z] 	/home/jenkins/agent/workspace/tidb_ghpr_unit_test/go/src/github.com/pingcap/tidb/ddl/ddl_tiflash_test.go:100 +0x59
[2022-02-10T10:19:34.685Z] reflect.Value.call(0x4055b40, 0xc000dbe9c0, 0x1213, 0x41b95a6, 0x4, 0xc006453f28, 0x1, 0x1, 0x67dc320, 0xc0004c76b8, ...)
[2022-02-10T10:19:34.685Z] 	/usr/local/go/src/reflect/value.go:476 +0x8e7
[2022-02-10T10:19:34.685Z] reflect.Value.Call(0x4055b40, 0xc000dbe9c0, 0x1213, 0xc0004c7728, 0x1, 0x1, 0xc0004c7728, 0x13afe1b, 0x1ac9aeb9c38530)
[2022-02-10T10:19:34.685Z] 	/usr/local/go/src/reflect/value.go:337 +0xb9
[2022-02-10T10:19:34.685Z] github.com/pingcap/check.(*suiteRunner).runFixture.func1(0xc0062d73b0)
[2022-02-10T10:19:34.685Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:799 +0x145
[2022-02-10T10:19:34.685Z] github.com/pingcap/check.(*suiteRunner).forkCall.func1(0xc000dc1c20, 0xc0062d73b0, 0x4319500)
[2022-02-10T10:19:34.685Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:739 +0xbf
[2022-02-10T10:19:34.685Z] created by github.com/pingcap/check.(*suiteRunner).forkCall
[2022-02-10T10:19:34.685Z] 	/go/pkg/mod/github.com/pingcap/check@v0.0.0-20211026125417-57bd13f7b5f0/check.go:734 +0x233

4. What is your TiDB version? (Required)

master fe0f1ea