{"page_status":{"page":{"name":"GitHub","subdomain":"github3","domain":"www.githubstatus.com","created_at":"2017-01-31T20:01:46.612Z","updated_at":"2024-10-04T07:52:38.355Z","url":"https://github.com","hidden_from_search":false,"css_body_background_color":"ffffff","css_font_color":"24292e","css_light_font_color":"6a737d","css_greens":"28a745","css_yellows":"dbab09","css_oranges":"e36209","css_reds":"dc3545","allow_page_subscribers":true,"allow_incident_subscribers":true,"notifications_from_email":"GitHub Status ","allow_email_subscribers":true,"allow_sms_subscribers":true,"twitter_username":"githubstatus","branding":"basic","support_url":"https://github.com/support","allow_webhook_subscribers":true,"css_border_color":"e1e4e8","css_graph_color":"0366d6","css_link_color":"0366d6","page_description":null,"activity_score":3032,"headline":null,"viewers_must_be_team_members":false,"ip_filters":null,"css_blues":"0366d6","time_zone":"UTC","notifications_reply_to_email":null,"notifications_email_footer":"You received this email because you are subscribed to GitHub's service status notifications.","allow_rss_atom_feeds":true,"black_hole":null,"over_allocations_cohort":null,"over_allocations_resolved_at":null,"custom_components_limit":null,"allow_slack_subscribers":true,"css_no_data":"b3bac5","deleted_at":null,"allow_teams_subscription":false,"max_maintenance_automation_allowed":null,"hero_cover":{"updated_at":null,"original_url":"","size":null,"normal_url":"","retina_url":""},"transactional_logo":{"updated_at":"2018-02-05T16:24:16.000+00:00","original_url":"//dka575ofm4ao0.cloudfront.net/pages-transactional_logos/original/36420/hTgCmUjbT7WMYBbAnxDp","size":7056,"normal_url":"//dka575ofm4ao0.cloudfront.net/pages-transactional_logos/normal/36420/hTgCmUjbT7WMYBbAnxDp","retina_url":"//dka575ofm4ao0.cloudfront.net/pages-transactional_logos/retina/36420/hTgCmUjbT7WMYBbAnxDp"},"favicon_logo":{"updated_at":"2018-01-17T23:38:34.000+00:00","size":6518,"url":"//dka575ofm4ao0.cloudfront.net/pages-favicon_logos/original/36420/akacZEQQfOBdc7ftyxJt"},"email_logo":{"updated_at":null,"original_url":"","size":null,"normal_url":"","retina_url":""},"twitter_logo":{"updated_at":"2019-01-10T19:02:40.000+00:00","size":4268,"url":"//dka575ofm4ao0.cloudfront.net/pages-twitter_logos/original/36420/GitHub-Mark-120px-plus.png"},"id":"kctbh9vrtdwd","organization_id":"30gcnw2xxnb0"}},"components":[{"status":"operational","name":"Git Operations","created_at":"2017-01-31T20:05:05.370Z","updated_at":"2024-09-25T16:03:30.193Z","position":1,"description":"Performance of git clones, pulls, pushes, and associated operations","showcase":false,"start_date":null,"id":"8l4ygp009s5s","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Webhooks","created_at":"2019-11-13T18:00:24.256Z","updated_at":"2024-09-13T07:13:24.599Z","position":2,"description":"Real time HTTP callbacks of user-generated and system events","showcase":false,"start_date":null,"id":"4230lsnqdsld","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Visit www.githubstatus.com for more information","created_at":"2018-12-05T19:39:40.838Z","updated_at":"2024-08-15T00:20:14.813Z","position":3,"description":null,"showcase":false,"start_date":null,"id":"0l2p9nhqnxpd","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"API Requests","created_at":"2017-01-31T20:01:46.621Z","updated_at":"2024-08-15T00:30:01.709Z","position":4,"description":"Requests for GitHub APIs","showcase":false,"start_date":null,"id":"brv1bkgrwx7q","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Issues","created_at":"2017-01-31T20:01:46.638Z","updated_at":"2024-09-13T07:13:24.616Z","position":5,"description":"Requests for Issues on GitHub.com","showcase":false,"start_date":null,"id":"kr09ddfgbfsf","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Pull Requests","created_at":"2020-09-02T15:39:06.329Z","updated_at":"2024-09-14T22:43:03.345Z","position":6,"description":"Requests for Pull Requests on GitHub.com","showcase":false,"start_date":null,"id":"hhtssxt0f5v2","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Actions","created_at":"2019-11-13T18:02:19.432Z","updated_at":"2024-09-25T19:19:01.226Z","position":7,"description":"Workflows, Compute and Orchestration for GitHub Actions","showcase":false,"start_date":null,"id":"br0l2tvcx85d","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Packages","created_at":"2019-11-13T18:02:40.064Z","updated_at":"2024-08-15T00:20:37.978Z","position":8,"description":"API requests and webhook delivery for GitHub Packages","showcase":false,"start_date":null,"id":"st3j38cctv9l","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Pages","created_at":"2017-01-31T20:04:33.923Z","updated_at":"2024-09-25T19:19:01.237Z","position":9,"description":"Frontend application and API servers for Pages builds","showcase":false,"start_date":null,"id":"vg70hn9s2tyj","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Codespaces","created_at":"2021-08-11T16:02:09.505Z","updated_at":"2024-09-30T11:26:49.242Z","position":10,"description":"Orchestration and Compute for GitHub Codespaces","showcase":false,"start_date":"2021-08-11T00:00:00.000Z","id":"h2ftsgbw7kmk","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Copilot","created_at":"2022-06-21T16:04:33.017Z","updated_at":"2024-09-26T05:03:53.133Z","position":11,"description":null,"showcase":false,"start_date":"2022-06-21T00:00:00.000Z","id":"pjmpxvq2cmr2","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false}],"months":[{"name":"October","year":2024,"starts_on":2,"days":31,"incidents":[]},{"name":"September","year":2024,"starts_on":0,"days":30,"incidents":[{"code":"bdzwt47fwg51","name":"Incident with Codespaces","message":"This incident has been resolved.","impact":"major","timestamp":"Sep 30, 11:08 - 11:26 UTC"},{"code":"wlb83pxg009y","name":"Disruption with some GitHub services","message":"On September 27th, 2024 from 10:26 to 10:34 UTC, some users listing project releases may have encountered a 500 error.","impact":"none","timestamp":"Sep 27, 15:30 - 15:30 UTC"},{"code":"zp4gzrqfzhrq","name":"Degraded performance for some Copilot users","message":"Between September 25, 2024, 22:20 UTC and September 26, 2024, 5:00 UTC the Copilot service was degraded. During this time Copilot chat requests failed at an average rate of 15%.

This was due to a faulty deployment in a service provider that caused server errors from multiple regions. Traffic was routed away from those regions at 22:28 UTC and 23:39 UTC, which partially restored functionality, while the upstream service provider rolled back their change. The rollback was completed at 04:41 UTC.

We are continuing to improve our ability to respond more quickly to similar issues through faster regional redirection and working with our upstream provider on improved monitoring.
","impact":"minor","timestamp":"Sep 25, 23:39 - Sep 26, 05:08 UTC"},{"code":"1g9v7rry4z86","name":"Incident with Actions Runs","message":"On September 25th, 2024 from 18:32 UTC to 19:13 UTC, Actions service experienced a degradation during a production deployment, leading to actions failing to be downloaded at the start of a job. On average, 21% of Actions workflow runs failed to start during the course of the incident. The issue was traced back to a bug in an internal service responsible for generating the URLs used by the Actions runner to download actions.

To mitigate the impact, we rolled back the affecting deployment. We are implementing new monitors to improve our detection and response time for this class of issues in the future.","impact":"minor","timestamp":"Sep 25, 19:11 - 19:19 UTC"},{"code":"q3xqwmcxzkqq","name":"Incident with Git Operations","message":"On September 25, 2024 from 14:31 UTC to 15:06 UTC the Git Operations service experienced a degradation, leading to 1,381,993 failed git operations. The overall error rate during this period was 4.2%, with a peak error rate of 12.5%.

The root cause was traced to a bug in a build script for a component that runs on the file servers that host git repository data. The build script incurred an error that did not cause the overall build process to fail, resulting in a faulty set of artifacts being deployed to production.

To mitigate the impact, we rolled back the affecting deployment.

To prevent further occurrences of this cause in the future, we will be addressing the underlying cause of the ignored build failure and improving metrics and alerting for the resulting production failure scenarios.","impact":"minor","timestamp":"Sep 25, 15:25 - 16:03 UTC"},{"code":"2dxdzc3fdgdx","name":"Incident with Codespaces start and creation","message":"On September 24th, 2024 from 08:20 UTC to 09:04 UTC the Codespaces service experienced an interruption in network connectivity, leading to 175 codespaces being unable to be created or resumed. The overall error rate during this period was 25%.

The cause was traced to an interruption in network connectivity caused by SNAT port exhaustion following a deployment, causing individual Codespaces to lose their connection to the service.

To mitigate the impact, we increased port allocations to give enough buffer for increased outbound connections shortly after deployments, and will be scaling up our outbound connectivity in the near future, as well as adding improved monitoring of network capacity to prevent future regressions.","impact":"major","timestamp":"Sep 24, 20:54 - 21:04 UTC"},{"code":"69sb0f8lydp4","name":"Incident with Pages and Actions","message":"On September 16, 2024, between 21:11 UTC and 22:20 UTC, Actions and Pages services were degraded. Customers who deploy Pages from a source branch experienced delayed runs. Approximately 1,100 runs were delayed long enough to get marked as abandoned. The runs that weren't abandoned completed successfully after we recovered from the incident. Actions jobs experienced average delays of 23 minutes, with some jobs experiencing delays as high as 45 minutes. During the course of the incident, 17% of runs were delayed by more than 5 minutes. At peak, as many as 80% of runs experienced delays exceeding 5 minutes. The root cause was a misconfiguration in the service that manages runner connections, which caused CPU throttling and led to a performance degradation in that service.

We mitigated the incident by diverting runner connections away from the misconfigured nodes. We are working to improve our internal monitoring and alerting to reduce our time to detection and mitigation of issues like this one in the future.","impact":"major","timestamp":"Sep 16, 21:31 - 22:08 UTC"},{"code":"r3x7x31k7nn1","name":"Disruption with Git SSH","message":"On September 16, 2024, between 13:24 UTC and 14:28 UTC, the Git Operations service experienced a degradation, leading to intermittent SSH connection drops. The overall SSH error rate during this period was 0.0005%, with a peak error rate of 0.3%. The root cause was traced to a regression in the service reload mechanism, which resulted in SSH hosts dropping connections on an hourly basis. As SSH hosts were rebooted for routine security updates, the issue progressively affected more hosts. To mitigate the impact, we removed the affected hosts from production traffic. The SSH regression has since been identified and resolved, with all SSH hosts fully restored. Additionally, we have implemented new monitoring to alert us of any SSH connection refusals moving forward.","impact":"minor","timestamp":"Sep 16, 13:29 - 14:28 UTC"},{"code":"rzqy51mc72dx","name":"Incident with Pull Requests","message":"On September 14th, 2024 from 20:45 UTC to 22:31 UTC commit creation operations, most commonly Pull Request merges, failed for some repositories. 226 repositories were impacted.

The root cause was a hardware fault in a Git file server, where merge commits are calculated. To mitigate the issue we marked the file server as offline.

Detection was slower than is typical because of lower weekend traffic. We’re making improvements to monitoring to decrease time to detection in future.","impact":"minor","timestamp":"Sep 14, 22:10 - 22:43 UTC"},{"code":"v677w7xlvkss","name":"Processing delays to some Issues, Pull Requests and Webhooks","message":"On Sep 13, 2024, between 05:03 UTC and 07:13 UTC, the Webhooks and Actions services were degraded resulting in some customers experiencing delayed processing of Webhooks and Actions Runs. 0.5% of Webhook deliveries were delayed more than 2 minutes during the incident. 15% of Actions Runs started between 05:03 and 05:24 UTC saw run start delays or failures. At 05:24 UTC, we implemented a mitigation to shift traffic to healthy infrastructure and new Actions Runs resumed normal operations. During the rest of the incident window, Actions runs started before 05:24 UTC continued to see delays publishing logs or job results. No Actions runs or Webhook deliveries were lost, only delayed.

We mitigated the incident by immediately shifting traffic to a healthy cluster while investigating. The incident was caused by an erroneous configuration change on our eventing platform. A permanent fix was deployed at 06:22 UTC after which services began to recover and burn down their backed up queues, with full recovery by 07:13 UTC.

We are working to reduce our time to detection and develop test automation to prevent issues like this one in the future.
","impact":"minor","timestamp":"Sep 13, 05:42 - 07:13 UTC"},{"code":"9dyp7zt86rp5","name":"Disruption with some GitHub services","message":"Between August 27, 2024, 15:52 UTC and September 5, 2024, 17:26 UTC the GitHub Connect service was degraded. This specifically impacted GHES customers who were enabling GitHub Connect for the first time on a GHES instance. Previously enabled GitHub Connect GHES instances were not impacted by this issue.

Customers experiencing this issue would have received a 404 response during GitHub Connect enablement and subsequent messages about a failure to connect. This was due to a recent change in configuration to GitHub Connect which has since been rolled back.

Subsequent enablement failures on re-attempts were caused by data corruption which has been remediated. Customers should now be able to enable GitHub Connect successfully.

To reduce our time to detection and mitigation of such issues in the future, we are working to improve observability of GitHub Connect failures. We are also making efforts to prevent future misconfiguration of GitHub Connect.","impact":"minor","timestamp":"Sep 5, 15:55 - 17:24 UTC"}]},{"name":"August","year":2024,"starts_on":4,"days":31,"incidents":[{"code":"djwfzglh2dyq","name":"Disruption with some GitHub services","message":"On August 29th, 2024, from 16:56 UTC to 21:42 UTC, we observed an elevated rate of traffic on our public edge, which triggered GitHub’s rate limiting protections. This resulted in <0.1% of users being identified as false-positives, which they experienced as intermittent connection timeouts. At 20:59 UTC the engineering team improved the system to remediate the false-positive identification of user traffic, and return to normal traffic operations.","impact":"minor","timestamp":"Aug 29, 19:29 - 21:54 UTC"},{"code":"345t4sjty6q3","name":"Disruption with some GitHub services","message":"On August 28, 2024, from 21:40 to 23:43 UTC, up to 25% of unauthenticated dotcom traffic in SE Asia (representing <1% of global traffic) encountered HTTP 500 errors. We observed elevated error rates at one of our global points of presence, where geo-DNS health checks were failing. We identified unhealthy cloud hardware in the region, indicated by abnormal CPU utilization patterns. As a result, we drained the site at 23:26 UTC, which promptly restored normal traffic operations.","impact":"minor","timestamp":"Aug 28, 22:02 - 23:43 UTC"},{"code":"1zyrr8fxrl21","name":"Disruption with some GitHub services","message":"On August 28th, 2024, starting at 20:43 UTC, some customers accessing GitHub from North America experienced degraded access to GitHub services. The error was intermittent and manifested as timeouts when requests tried to reach endpoints. This was due to a degraded route internal to one of our transit providers. We identified the unhealthy provider path and drained it at 23:26 UTC, rerouting traffic through other providers and promptly restoring normal traffic operations.","impact":"minor","timestamp":"Aug 27, 22:37 - 23:26 UTC"},{"code":"30jg3tzwbv93","name":"Incident with Actions","message":"On August 22, 2024, between 16:10 UTC and 17:28 UTC, Actions experienced degraded performance leading to failed workflow runs. On average, 2.5% of workflow runs failed to start with the failure rate peaking at 6%. In addition we saw a 1% error rate for Actions API endpoints. This was due to an Actions service being deployed to faulty hardware that had an incorrect memory configuration, leading to significant performance degradation of those pods due to insufficient memory.

The impact was mitigated when the pods were evicted automatically and moved to healthy hosts. The faulty hardware was disabled to prevent a recurrence. We are improving our health checks to ensure that unhealthy hardware is consistently marked offline automatically. We are also improving our monitoring and deployment practices to reduce our time to detection and automated mitigation at the service layer for issues like this in the future.","impact":"minor","timestamp":"Aug 22, 16:49 - 17:28 UTC"},{"code":"3vhw289ftnqp","name":"Incident with Actions","message":"On August 21, 2024, between 13:48 UTC and 15:00 UTC, Actions experienced degraded performance, leading to delays in workflow runs. On average, 25% of workflow runs were delayed by 8 minutes. Less than 1% of workflow runs exhausted retries and failed to start. The issue stemmed from a backlog of Pull Request events which caused delays in Actions processing the event queues that trigger workflow runs.

We mitigated the incident by disabling the process that led to the sudden spike in Pull Request events. We are working to improve our monitoring and deployment practices to reduce our time to detection and mitigation of issues like this one in the future. We are also identifying appropriate changes to rate limits and reserved capacity to reduce the breadth of impact.","impact":"minor","timestamp":"Aug 21, 14:09 - 15:11 UTC"},{"code":"mrpx4trfk45z","name":"Incident with starting Action Workflows","message":"On August 15, 2024, between 13:14 UTC and 13:43 UTC, the Actions service was degraded and resulted in failures to start new workflow runs for customers of github.com. On average, 10% of Actions workflow runs failed to start with the failure rate peaking at 15%. This was due to an infrastructure change that enabled a network proxy for requests between the Actions service and an internal API which caused requests to fail.

We mitigated the incident by rolling back the change. We are working to improve our monitoring and deployment practices to reduce our time to detection and mitigation of issues like this one in the future.","impact":"minor","timestamp":"Aug 15, 13:35 - 13:59 UTC"},{"code":"kz4khcgdsfdv","name":"All GitHub services are experiencing significant disruptions","message":"On August 14, 2024 between 23:02 UTC and 23:38 UTC, all GitHub services on GitHub.com were inaccessible for all users.

This was due to a configuration change that impacted traffic routing within our database infrastructure, resulting in critical services unexpectedly losing database connectivity. There was no data loss or corruption during this incident.

At 22:59 UTC an erroneous configuration change rolled out to all GitHub.com databases that impacted the ability of the database to respond to health check pings from the routing service. As a result, the routing service could not detect healthy databases to route application traffic to. This led to widespread impact on GitHub.com starting at 23:02 UTC.

We mitigated the incident by reverting the change and confirming restored connectivity to our databases. At 23:38 UTC, traffic resumed and all services recovered to full health. Out of an abundance of caution, we continued to monitor before resolving the incident at 00:30 UTC on August 15th, 2024.

To prevent recurrence we are implementing additional guardrails in our database change management process. We are also prioritizing several repair items such as faster rollback functionality and more resilience to dependency failures.

Given the severity of this incident, follow-up items are the highest priority work for teams at this time.","impact":"major","timestamp":"Aug 14, 23:11 - Aug 15, 00:30 UTC"},{"code":"2qxyr481ywxf","name":"Disruption with some GitHub services","message":"On August 13, 2024, between 13:00 UTC and 13:23 UTC the Copilot service and some parts of the GitHub UI were degraded. This impacted about 25% of GitHub.com users. This was due to a partial rollout of a caching layer for Copilot licensing checks. During the rollout, connections to the caching layer were overwhelmed causing the licensing checks to timeout. Many pages were impacted by this failure due to a lack of resiliency to the timeouts.

We mitigated the incident by reverting the rollout of the caching layer 11 minutes after initial detection. This immediately restored functionality for affected users.

We are working to gracefully degrade experiences during these types of failures and reduce dependencies across services that may cause these types of failures in the future.
","impact":"minor","timestamp":"Aug 13, 13:11 - 13:23 UTC"},{"code":"ndlzqycv0dh8","name":"Incident with Webhooks","message":"On August 12, 2024 from 13:39 to 14:28 UTC some users experienced an elevated rate of errors of up to 0.45% from the GitHub API. Less than 5% of webhooks interactions failed and less than 0.5% of Actions runs were delayed.

This impact was caused by internal networking instances being insufficiently scaled.

We mitigated the incident by provisioning additional instances. We are working to enhance the sizing strategy for the relevant infrastructure to prevent similar issues and to also improve monitoring and processes to reduce our time to detection and mitigation of issues like this one in the future.","impact":"minor","timestamp":"Aug 12, 14:09 - 14:41 UTC"},{"code":"fnj6l879tbtw","name":"Disruption with some GitHub services","message":"This incident has been resolved.","impact":"minor","timestamp":"Aug 12, 14:03 - 14:15 UTC"},{"code":"1kn4kywmvgz4","name":"[Retroactive] Incident with Pull Requests","message":"Beginning at 6:52 PM UTC on August 6, 2024 and lasting until 6:59 PM UTC, some customers of github.com saw errors when navigating to a Pull Request. The error rate peaked at ~5% for logged in users. This was due to a change which was rolled back after alerts fired during the deployment. We did not status before we completed the rollback, and the incident is currently resolved. We are sorry for the delayed post on githubstatus.com.","impact":"minor","timestamp":"Aug 6, 18:30 - 18:30 UTC"}]}],"show_component_filter":false,"show_uptime_calendar":true,"component_filter":null,"start_time":"2024-08-01T00:00:00Z","end_time":"2024-10-31T23:59:59Z"}

Sep	OCT	Nov
	05
2023	2024	2025

Incident History