Resolved -
On April 5, 2024, between 8:11 and 8:58 UTC a number of GitHub services were degraded, returning error responses. Web request error rate peaked at 6%, API request error rate peaked at 10%. Actions had 103,660 workflow runs fail to start.
A database load balancer change caused connection failures in one of our three data centers to various critical database clusters. The incident was mitigated once that change was rolled back.
We have updated our deployment pipeline to better detect this problem in earlier stages of rollout to reduce impact to end users.
Apr 5, 09:18 UTC
Update -
Pull Requests is operating normally.
Apr 5, 09:17 UTC
Update -
Issues is operating normally.
Apr 5, 09:17 UTC
Update -
API Requests is operating normally.
Apr 5, 09:17 UTC
Update -
Codespaces is operating normally.
Apr 5, 09:17 UTC
Update -
Actions is operating normally.
Apr 5, 09:17 UTC
Update -
Pages is operating normally.
Apr 5, 09:17 UTC
Update -
Actions is experiencing degraded performance. We are continuing to investigate.
Apr 5, 09:17 UTC
Update -
We've reverted a change we believe caused this, are seeing initial indications of reduced errors, and are monitoring for full recovery
Apr 5, 09:00 UTC
Update -
Pages is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:59 UTC
Update -
We're seeing connection failures to some databases in two of three sites and are investigating.
Apr 5, 08:51 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:50 UTC
Update -
Issues is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:49 UTC
Update -
API Requests is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:49 UTC
Update -
Codespaces is experiencing degraded performance. We are continuing to investigate.
Apr 5, 08:49 UTC
Investigating -
We are investigating reports of degraded availability for Actions
Apr 5, 08:33 UTC
