The Wayback Machine - https://web.archive.org/web/20220309031452/https://github.com/googleforgames/agones/issues/2479
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky: TestFleetRecreateGameServers/gameserver_shutdown #2479

Open
markmandel opened this issue Feb 14, 2022 · 1 comment
Open

Flaky: TestFleetRecreateGameServers/gameserver_shutdown #2479

markmandel opened this issue Feb 14, 2022 · 1 comment
Labels
area/tests good first issue help wanted kind/bug

Comments

@markmandel
Copy link
Member

@markmandel markmandel commented Feb 14, 2022

What happened:

It fails sometimes!

--- FAIL: TestFleetRecreateGameServers (0.00s)
    --- FAIL: TestFleetRecreateGameServers/gameserver_shutdown (89.80s)
        fleet_test.go:1160: Could not message GameServer: timed out attempting to send UDP packet to address: timed out waiting for the condition

What you expected to happen:

It to pass!

How to reproduce it (as minimally and precisely as possible):

Run the test lots.

Anything else we need to know?:

If I had to guess, the test is expecting a response, but for whatever reason that UDP packet is lost on the way back (which is never guaranteed). The condition for the test should be on the Shutdown of the GameServer, not on the receipt of the message.

Environment: Cloud Build

  • Agones version: dev
  • Kubernetes version (use kubectl version): 1.21.0
  • Cloud provider or hardware configuration: GKE
  • Install method (yaml/helm): helm
  • Troubleshooting guide log(s):
  • Others:
@markmandel
Copy link
Member Author

@markmandel markmandel commented Feb 14, 2022

Test logs:

Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:04.296" level=info msg="waiting for fleet condition" fleet=simple-fleet-rp9gw test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:06.889" level=info msg="Checking Fleet Ready replicas" expected=10 fleet=simple-fleet-rp9gw fleetStatus="{Replicas:10 ReadyReplicas:0 ReservedReplicas:0 AllocatedReplicas:0 Players:0xc00013e9d0}" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:10.468" level=info msg="Checking Fleet Ready replicas" expected=10 fleet=simple-fleet-rp9gw fleetStatus="{Replicas:10 ReadyReplicas:6 ReservedReplicas:0 AllocatedReplicas:0 Players:0xc0005b0f20}" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:13.468" level=info msg="Checking Fleet Ready replicas" expected=10 fleet=simple-fleet-rp9gw fleetStatus="{Replicas:10 ReadyReplicas:6 ReservedReplicas:0 AllocatedReplicas:0 Players:0xc00097ad50}" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:17.468" level=info msg="Checking Fleet Ready replicas" expected=10 fleet=simple-fleet-rp9gw fleetStatus="{Replicas:10 ReadyReplicas:10 ReservedReplicas:0 AllocatedReplicas:0 Players:0xc00098f620}" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:29.793" level=info msg="Could not read from address" address="35.197.104.56:7356" error="read udp 192.168.10.7:37285->35.197.104.56:7356: i/o timeout" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:31.834" level=info msg="Could not read from address" address="35.197.104.56:7356" error="read udp 192.168.10.7:39018->35.197.104.56:7356: read: connection refused" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:33.835" level=info msg="Could not read from address" address="35.197.104.56:7356" error="read udp 192.168.10.7:47245->35.197.104.56:7356: read: connection refused" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:01:35.834" level=info msg="Could not read from address" address="35.197.104.56:7356" error="read udp 192.168.10.7:60829->35.197.104.56:7356: read: connection refused" test=TestFleetRecreateGameServers/gameserver_shutdown
... repeated a bunch ...
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:25.834" level=info msg="Could not read from address" address="35.197.104.56:7356" error="read udp 192.168.10.7:37175->35.197.104.56:7356: read: connection refused" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:27.834" level=info msg="Could not read from address" address="35.197.104.56:7356" error="read udp 192.168.10.7:52583->35.197.104.56:7356: read: connection refused" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:29.834" level=info msg="Could not read from address" address="35.197.104.56:7356" error="read udp 192.168.10.7:41628->35.197.104.56:7356: read: connection refused" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:29.873" level=info msg="Could not read from address" address="35.197.104.56:7356" error="read udp 192.168.10.7:48637->35.197.104.56:7356: read: connection refused" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:29.874" level=info msg="Failed to send UDP packet to GameServer. Dumping Events!" gs=simple-fleet-rp9gw-x7j29-2xxnd status="{State:Ready Ports:[{Name:udp-port Port:7356}] Address:35.197.104.56 NodeName:gke-e2e-test-cluster-default-684ee24f-whtv ReservedUntil:<nil> Players:0xc0008c7800}" test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:29.874" level=info msg="Dumping Events:" kind=GameServer test=TestFleetRecreateGameServers/gameserver_shutdown
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:30.064" level=info msg="Event!" lastTimestamp="2022-02-14 18:01:04 +0000 UTC" message="Port allocated" reason=PortAllocation test=TestFleetRecreateGameServers/gameserver_shutdown type=Normal
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:30.065" level=info msg="Event!" lastTimestamp="2022-02-14 18:01:04 +0000 UTC" message="Pod simple-fleet-rp9gw-x7j29-2xxnd created" reason=Creating test=TestFleetRecreateGameServers/gameserver_shutdown type=Normal
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:30.065" level=info msg="Event!" lastTimestamp="2022-02-14 18:01:04 +0000 UTC" message="Address and port populated" reason=Scheduled test=TestFleetRecreateGameServers/gameserver_shutdown type=Normal
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:30.065" level=info msg="Event!" lastTimestamp="2022-02-14 18:01:08 +0000 UTC" message="SDK state change" reason=RequestReady test=TestFleetRecreateGameServers/gameserver_shutdown type=Normal
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:30.065" level=info msg="Event!" lastTimestamp="2022-02-14 18:01:08 +0000 UTC" message="SDK.Ready() complete" reason=Ready test=TestFleetRecreateGameServers/gameserver_shutdown type=Normal
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:30.065" level=info msg="Event!" lastTimestamp="2022-02-14 18:01:20 +0000 UTC" message="SDK state change" reason=Shutdown test=TestFleetRecreateGameServers/gameserver_shutdown type=Normal
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:30.065" level=info msg="Event!" lastTimestamp="2022-02-14 18:01:20 +0000 UTC" message="Deletion started" reason=Shutdown test=TestFleetRecreateGameServers/gameserver_shutdown type=Normal
Step #21 - "e2e-feature-gates": time="2022-02-14 18:02:30.065" level=info msg="Event!" lastTimestamp="2022-02-14 18:01:20 +0000 UTC" message="Deleting Pod simple-fleet-rp9gw-x7j29-2xxnd" reason=Shutdown test=TestFleetRecreateGameServers/gameserver_shutdown type=Normal
Step #21 - "e2e-feature-gates":     --- FAIL: TestFleetRecreateGameServers/gameserver_shutdown (89.80s)

We can see that the Shutdown does occur as expected. So I figure it's a failure in the test code.
@markmandel markmandel mentioned this issue Feb 14, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/tests good first issue help wanted kind/bug
1 participant
@markmandel