{"page_status":{"page":{"name":"GitHub","subdomain":"github3","domain":"www.githubstatus.com","created_at":"2017-01-31T20:01:46.612Z","updated_at":"2024-02-23T08:07:35.535Z","url":"https://github.com","hidden_from_search":false,"css_body_background_color":"ffffff","css_font_color":"24292e","css_light_font_color":"6a737d","css_greens":"28a745","css_yellows":"dbab09","css_oranges":"e36209","css_reds":"dc3545","allow_page_subscribers":true,"allow_incident_subscribers":true,"notifications_from_email":"GitHub Status ","allow_email_subscribers":true,"allow_sms_subscribers":true,"twitter_username":"githubstatus","branding":"basic","support_url":"https://github.com/support","allow_webhook_subscribers":true,"css_border_color":"e1e4e8","css_graph_color":"0366d6","css_link_color":"0366d6","page_description":null,"activity_score":2882,"headline":null,"viewers_must_be_team_members":false,"ip_filters":null,"css_blues":"0366d6","time_zone":"UTC","notifications_reply_to_email":null,"notifications_email_footer":"You received this email because you are subscribed to GitHub's service status notifications.","allow_rss_atom_feeds":true,"black_hole":null,"over_allocations_cohort":null,"over_allocations_resolved_at":null,"custom_components_limit":null,"allow_slack_subscribers":true,"css_no_data":"b3bac5","deleted_at":null,"allow_teams_subscription":null,"hero_cover":{"updated_at":null,"original_url":"","size":null,"normal_url":"","retina_url":""},"transactional_logo":{"updated_at":"2018-02-05T16:24:16.000+00:00","original_url":"//dka575ofm4ao0.cloudfront.net/pages-transactional_logos/original/36420/hTgCmUjbT7WMYBbAnxDp","size":7056,"normal_url":"//dka575ofm4ao0.cloudfront.net/pages-transactional_logos/normal/36420/hTgCmUjbT7WMYBbAnxDp","retina_url":"//dka575ofm4ao0.cloudfront.net/pages-transactional_logos/retina/36420/hTgCmUjbT7WMYBbAnxDp"},"favicon_logo":{"updated_at":"2018-01-17T23:38:34.000+00:00","size":6518,"url":"//dka575ofm4ao0.cloudfront.net/pages-favicon_logos/original/36420/akacZEQQfOBdc7ftyxJt"},"email_logo":{"updated_at":null,"original_url":"","size":null,"normal_url":"","retina_url":""},"twitter_logo":{"updated_at":"2019-01-10T19:02:40.000+00:00","size":4268,"url":"//dka575ofm4ao0.cloudfront.net/pages-twitter_logos/original/36420/GitHub-Mark-120px-plus.png"},"id":"kctbh9vrtdwd","organization_id":"30gcnw2xxnb0"}},"components":[{"status":"operational","name":"Git Operations","created_at":"2017-01-31T20:05:05.370Z","updated_at":"2024-02-05T09:53:13.382Z","position":1,"description":"Performance of git clones, pulls, pushes, and associated operations","showcase":false,"start_date":null,"id":"8l4ygp009s5s","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"API Requests","created_at":"2017-01-31T20:01:46.621Z","updated_at":"2024-01-09T14:24:13.860Z","position":2,"description":"Requests for GitHub APIs","showcase":false,"start_date":null,"id":"brv1bkgrwx7q","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Webhooks","created_at":"2019-11-13T18:00:24.256Z","updated_at":"2024-02-09T11:25:01.453Z","position":3,"description":"Real time HTTP callbacks of user-generated and system events","showcase":false,"start_date":null,"id":"4230lsnqdsld","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Visit www.githubstatus.com for more information","created_at":"2018-12-05T19:39:40.838Z","updated_at":"2022-09-07T00:08:33.519Z","position":4,"description":null,"showcase":false,"start_date":null,"id":"0l2p9nhqnxpd","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Issues","created_at":"2017-01-31T20:01:46.638Z","updated_at":"2024-01-09T14:24:02.127Z","position":5,"description":"Requests for Issues on GitHub.com","showcase":false,"start_date":null,"id":"kr09ddfgbfsf","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Pull Requests","created_at":"2020-09-02T15:39:06.329Z","updated_at":"2024-01-09T14:38:57.170Z","position":6,"description":"Requests for Pull Requests on GitHub.com","showcase":false,"start_date":null,"id":"hhtssxt0f5v2","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Actions","created_at":"2019-11-13T18:02:19.432Z","updated_at":"2024-02-21T17:30:06.927Z","position":7,"description":"Workflows, Compute and Orchestration for GitHub Actions","showcase":false,"start_date":null,"id":"br0l2tvcx85d","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Packages","created_at":"2019-11-13T18:02:40.064Z","updated_at":"2024-01-09T14:37:18.000Z","position":8,"description":"API requests and webhook delivery for GitHub Packages","showcase":false,"start_date":null,"id":"st3j38cctv9l","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Pages","created_at":"2017-01-31T20:04:33.923Z","updated_at":"2024-01-09T14:16:28.324Z","position":9,"description":"Frontend application and API servers for Pages builds","showcase":false,"start_date":null,"id":"vg70hn9s2tyj","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Codespaces","created_at":"2021-08-11T16:02:09.505Z","updated_at":"2024-01-21T09:34:34.651Z","position":10,"description":"Orchestration and Compute for GitHub Codespaces","showcase":false,"start_date":"2021-08-11T00:00:00.000Z","id":"h2ftsgbw7kmk","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false},{"status":"operational","name":"Copilot","created_at":"2022-06-21T16:04:33.017Z","updated_at":"2024-02-12T18:14:07.190Z","position":11,"description":null,"showcase":false,"start_date":"2022-06-21T00:00:00.000Z","id":"pjmpxvq2cmr2","group_id":null,"page_id":"kctbh9vrtdwd","group":false,"only_show_if_degraded":false}],"months":[{"name":"February","year":2024,"starts_on":4,"days":29,"incidents":[{"code":"wn6s1w8vkk1y","name":"Incident with Actions","message":"This incident has been resolved.","impact":"minor","timestamp":"Feb 21, 17:20 - 17:30 UTC"},{"code":"k0qjh7s64fxk","name":"Incident with Copilot","message":"On Monday February 12th, 2024, 03:00 UTC we deployed a code change to a component of Copilot. At 06:00 UTC we observed an increase in timeouts for code completions impacting 55% of Copilot users at peak across Asia and Europe.

At 12:00 UTC we restarted the nodes, and response durations returned to normal operation until 13:00 UTC when response durations degraded again. At 16:15 UTC we made a configuration change to send traffic to regions that were not exhibiting the errors, which resulted in code completions working fully although completing at a higher latency than normal for some users. At 18:00 UTC we reverted the deploy and response durations returned to normal.

We have added better monitoring to components that failed to decrease resolution times to incidents like this in the future.","impact":"minor","timestamp":"Feb 12, 13:28 - 18:14 UTC"},{"code":"vv6z7v5w80yr","name":"Incident with Copilot","message":"On Monday February 12th, 2024, 03:00 UTC we deployed a code change to a component of Copilot. At 06:00 UTC we observed an increase in timeouts for code completions impacting 55% of Copilot users at peak across Asia and Europe.

At 12:00 UTC we restarted the nodes, and response durations returned to normal operation until 13:00 UTC when response durations degraded again. At 16:15 UTC we made a configuration change to send traffic to regions that were not exhibiting the errors, which resulted in code completions working fully although completing at a higher latency than normal for some users. At 18:00 UTC we reverted the deploy and response durations returned to normal.

We have added better monitoring to components that failed to decrease resolution times to incidents like this in the future.","impact":"minor","timestamp":"Feb 12, 11:38 - 12:39 UTC"},{"code":"my7r3l9lsqnk","name":"Incident with Webhooks","message":"On February 9, 2024 between 10:34 UTC and 11:24 UTC, the Webhooks service was degraded and 63% of webhooks were delayed by up to 16 minutes with an average delay of 5 minutes. No webhook deliveries were lost. This was due to an issue with an overloaded backend data store that was unable to process network requests fast enough.

We mitigated the incident by manually failing over traffic to healthy hosts.

We are expanding the capacity of the backing store as well as making the Webhooks service more resilient to this kind of issue.
","impact":"minor","timestamp":"Feb 9, 11:09 - 11:28 UTC"},{"code":"3k40h28fkb2r","name":"Incident with Git Operations","message":"On 2024-02-05, from 09:26 to 13:20 UTC some GitHub customers experienced errors when trying to download raw files. An overloaded server exposed a bug, causing us to return HTTP 500 error codes.

The issue was mitigated by disabling the server and re-routing traffic. We are implementing improvements to our routing logic to more quickly avoid troublesome hosts in the future.
","impact":"minor","timestamp":"Feb 5, 09:40 - 09:53 UTC"},{"code":"7k5wl5j4f1t4","name":"We are investigating reports of degraded performance.","message":"An update to our design system caused issues loading dynamic content in the global side navigation menu and in other page-specific sidebar navigation elements. Impacted users saw continuous loading spinners in place of dynamic menu content. User impact lasted from 0:55 UTC to 4:41 UTC on February 1st.

We are working on a number of improvements in response to this incident. We are adding request volume monitors to sidebar navigation endpoints and making changes to our front end escalation paths to improve our time to detect and time to recovery for incidents of this nature. We have also begun work to improve both automated and manual testing for these types of changes in order to prevent recurrence.","impact":"minor","timestamp":"Feb 1, 03:13 - 04:41 UTC"}]},{"name":"January","year":2024,"starts_on":1,"days":31,"incidents":[{"code":"5y8b8lsqbbyq","name":"We are investigating reports of degraded performance.","message":"This incident was the result of an infrastructure change that was made to our load balancers to prepare us for IPv6 enablement of GitHub.com. This change was deployed to a subset of our global edge sites.

The change had the unintended consequence of causing IPv4 addresses to start being passed as an IPv4-mapped IPv6-compatible address to our IP Allow List functionality.

For example 10.1.2.3 became ::ffff:10.1.2.3. While our IP Allow List functionality was developed with IPv6 in mind, it wasn't developed to handle these mapped addresses, and hence started blocking requests as it deemed these to be not in the defined list of allowed addresses. Request error rates peaked at 0.23% of all requests.

We have so far identified three remediation items here:

- Update the IP Allow List functionality to handle IPv4-mapped addresses.
- Audit the rest of our stack to confirm there are no further places this IPv4-mapped IPv6 addresses flaw exists.
- Improve our testing and monitoring processes to better catch these issues in the future.","impact":"major","timestamp":"Jan 31, 14:14 - 14:57 UTC"},{"code":"g6drnqm54qd4","name":"We are investigating reports of degraded performance.","message":"On January 28, 2024, between 01:00 UTC and 14:00 UTC the Avatars service was degraded and could not return all avatar images requested by users, instead it would return a default, fallback avatar image. This incident impacted, at peak time 6% of the requests for viewing Avatars. Requests that were impacted did not prevent the users from continuing to use any GitHub services. This was due to an issue with the Avatars service connecting to a database host.

We mitigated the incident by restarting the malfunctioning hosts that were not able to return the user avatar images.

We are working to improve alerting and monitoring of our services to reduce our time to detection and mitigation.","impact":"minor","timestamp":"Jan 28, 13:20 - 14:42 UTC"},{"code":"bsk6hmj0d7nv","name":"We are investigating reports of degraded performance.","message":"On January 23, 2024 at 14:36 UTC, our internal metrics began showing an increase in exceptions originating from our live update service. Live updates to Issues, PRs, Actions, and Projects were failing, but refreshing the page successfully updated page content. We resolved the issue by rolling back a problematic dependency update and reenabled live updates at 18:53 UTC.

We are working to improve alerting and monitoring of our live update service to reduce our time to detection and mitigation.","impact":"major","timestamp":"Jan 23, 15:50 - 18:53 UTC"},{"code":"7ck5966p1073","name":"Incident with Codespaces","message":"On 2024-01-21 at 3:38 UTC, we experienced an incident that affected customers using Codespaces. Customers encountered issues creating and resuming Codespaces in multiple regions due to operational issues with compute and storage resources.

Around 25% of customers were impacted, primarily in East US and West Europe. We re-routed traffic for Codespace creations to less impacted regions, but existing Codespaces in these regions may have been unable to resume during the incident.

By 7:30 UTC, we had recovered connectivity to all regions except West Europe, which had an extended recovery time due to increased load in that particular region. The incident was resolved on 2024-01-21 at 9:34 UTC once Codespace creations and resumes were working normally in all regions.","impact":"major","timestamp":"Jan 21, 03:38 - 09:34 UTC"},{"code":"hmvr5kpgzc45","name":"Incident with Actions","message":"On 2024-01-21 from 02:05 UTC to 06:19 UTC, GitHub Hosted Runners experienced increased error rates from our main cloud service provider. The errors were initially limited to a single region and we were able to route around the issue by transparently failing over to other regions. However, errors gradually expanded across all regions we deploy to and led to our available compute capacity being exhausted.

During the incident, up to 35% of Actions jobs using Larger Runners and 2% of Actions jobs using GitHub Hosted Runners overall may have experienced intermittent delays in starting. Once the issue was resolved by our cloud service provider, our systems made a full recovery without intervention.

We’re working closely with our service provider to understand the cause of the outage and mitigations we can put in place. We’re also working to increase our resilience to outages of this nature by expanding the regions we deploy to beyond the existing set, especially for Larger Runners.","impact":"minor","timestamp":"Jan 21, 03:45 - 06:19 UTC"},{"code":"pxg3dz4yg7lp","name":"Incident with Issues, API Requests, Pull Requests, Actions, Pages, Git Operations, Webhooks, Packages and Codespaces","message":"On January 9 between 12:45 and 13:56 UTC, services in one of our three sites experienced elevated latency for connections. This led to a sustained period of timed out requests across a number of services, including but not limited to our git backend. An average of 5% and max of 10% of requests failed with a 5xx response or timed out during this period. This was caused by a combination of events that led to connection limits being hit in load balancer proxies in that site. An upgrade of hosts was in flight, which meant a subset of proxy hosts were draining and coming offline as the upgrade rolled through the fleet. A config change event also triggered a connection reset across all services in that site. These events are commonplace, but led to a spike in connection establishment events that led to the online proxy hosts hitting the connection limit. Upon further analysis, that limit was lower than it should have been. We have increased that limit to prevent this from recurring. We have also identified improvements to our monitoring of connection limits and behavior and changes to reduce the risk of proxy host upgrades leading to reduced capacity.","impact":"major","timestamp":"Jan 9, 13:02 - 14:40 UTC"},{"code":"dxthh9653wkk","name":"We are investigating reports of degraded performance.","message":"On January 9 between 1:06 and 5:43 UTC, no audit log events were streamed for customers that have that enabled. All events that happened during this time were delivered after the issue was mitigated. The event delivery was failing due to a data shape issue, with a streaming configuration linked to a soft-deleted Enterprise causing a runtime error for a backend service. This was mitigated by removing that configuration. Since the incident, we have improved our detection of these errors and ensured support for similar scenarios.","impact":"minor","timestamp":"Jan 9, 04:59 - 05:44 UTC"},{"code":"35pgg0gc75nv","name":"Incident with Codespaces","message":"From January 5th to January 8th, Codespaces experienced issues with port forwarding when connecting from a web browser. During this incident 100% of operations to forward ports and connect to a forwarded port failed for customers located in US West and Australia. The cause was due to a cross origin API change in our port forwarding service. After detecting the issue, we mitigated the impact by rolling back the change that caused the discrepancy in the cross origin rules. We have improved the way we are detecting issues like this one including implementing automated alerts during and after the rollout process so that we get signals as early as possible.","impact":"minor","timestamp":"Jan 8, 20:22 - 23:41 UTC"},{"code":"zzy7pdshgqk0","name":"We are investigating reports of degraded performance.","message":"From 10:30 to 16:15 UTC on January 3rd, our `/settings/emails` page experienced downtime due to an outage in our contact permissions system. During this time customers were unable to update their email subscription preferences.

This system plays a crucial role in managing email subscriptions and the contactability of GitHub customers by our marketing and sales teams. Unfortunately, the outage prevented the generation of links for managing email subscriptions, resulting in the `/settings/emails` page timing out. Consequently, customers were unable to update their email subscription preferences during this period.

During the incident we implemented graceful degradation on the `/settings/email` page to enable certain functionality, albeit with slower response times. By approximately 16:15 UTC on January 3rd, our contact permissions system was fully operational again, and customers could manage their email subscription preferences as usual.

With the issue now resolved, our team is enhancing the `/settings/emails` page resilience to minimize the impact of system outages in the future. This includes better request timeout handling and improved metrics and monitoring in this area of the application.","impact":"minor","timestamp":"Jan 3, 12:42 - 17:05 UTC"}]},{"name":"December","year":2023,"starts_on":5,"days":31,"incidents":[{"code":"1ksntt4bk3bq","name":"We are investigating reports of degraded performance.","message":"On December 26, 2023, GitHub received a report through our Bug Bounty Program demonstrating a vulnerability which, if exploited, allowed access to credentials within a production container. We fixed this vulnerability on GitHub.com the same day and began rotating all potentially exposed credentials. Through this process we found some flaws in how we rotate certain credentials and are working on improving our credential rotation process. More detail can be found on our blog: https://github.blog/2024-01-16-rotating-credentials-for-github-com-and-new-ghes-patches/","impact":"minor","timestamp":"Dec 29, 20:05 - 21:21 UTC"},{"code":"464b7g25n77j","name":"Incident with API Requests","message":"On December 26, 2023, GitHub received a report through our Bug Bounty Program demonstrating a vulnerability which, if exploited, allowed access to credentials within a production container. We fixed this vulnerability on GitHub.com the same day and began rotating all potentially exposed credentials. Through this process we found some flaws in how we rotate certain credentials and are working on improving our credential rotation process. More detail can be found on our blog: https://github.blog/2024-01-16-rotating-credentials-for-github-com-and-new-ghes-patches/","impact":"minor","timestamp":"Dec 29, 18:17 - 18:33 UTC"},{"code":"1h37j1mb9xnx","name":"We are investigating reports of degraded performance.","message":"On December 26, 2023, GitHub received a report through our Bug Bounty Program demonstrating a vulnerability which, if exploited, allowed access to credentials within a production container. We fixed this vulnerability on GitHub.com the same day and began rotating all potentially exposed credentials. Through this process we found some flaws in how we rotate certain credentials and are working on improving our credential rotation process. More detail can be found on our blog: https://github.blog/2024-01-16-rotating-credentials-for-github-com-and-new-ghes-patches/","impact":"minor","timestamp":"Dec 29, 01:41 - 02:04 UTC"},{"code":"lt4kyp4tgx45","name":"We are investigating reports of degraded performance.","message":"On December 26, 2023, GitHub received a report through our Bug Bounty Program demonstrating a vulnerability which, if exploited, allowed access to credentials within a production container. We fixed this vulnerability on GitHub.com the same day and began rotating all potentially exposed credentials. Through this process we found some flaws in how we rotate certain credentials and are working on improving our credential rotation process. More detail can be found on our blog: https://github.blog/2024-01-16-rotating-credentials-for-github-com-and-new-ghes-patches/","impact":"major","timestamp":"Dec 28, 06:43 - 06:57 UTC"},{"code":"hzd6d6xhjpl6","name":"Incident with Pages","message":"On December 26, 2023, GitHub received a report through our Bug Bounty Program demonstrating a vulnerability which, if exploited, allowed access to credentials within a production container. We fixed this vulnerability on GitHub.com the same day and began rotating all potentially exposed credentials. Through this process we found some flaws in how we rotate certain credentials and are working on improving our credential rotation process. More detail can be found on our blog: https://github.blog/2024-01-16-rotating-credentials-for-github-com-and-new-ghes-patches/","impact":"minor","timestamp":"Dec 27, 18:29 - 19:25 UTC"},{"code":"rp7669f45fjw","name":"Incident with Codespaces","message":"On December 26, 2023, GitHub received a report through our Bug Bounty Program demonstrating a vulnerability which, if exploited, allowed access to credentials within a production container. We fixed this vulnerability on GitHub.com the same day and began rotating all potentially exposed credentials. Through this process we found some flaws in how we rotate certain credentials and are working on improving our credential rotation process. More detail can be found on our blog: https://github.blog/2024-01-16-rotating-credentials-for-github-com-and-new-ghes-patches/","impact":"major","timestamp":"Dec 27, 02:53 - 04:00 UTC"},{"code":"xlm9bd5xq5bk","name":"Incident with Codespaces and API Requests","message":"On December 26, 2023, GitHub received a report through our Bug Bounty Program demonstrating a vulnerability which, if exploited, allowed access to credentials within a production container. We fixed this vulnerability on GitHub.com the same day and began rotating all potentially exposed credentials. Through this process we found some flaws in how we rotate certain credentials and are working on improving our credential rotation process. More detail can be found on our blog: https://github.blog/2024-01-16-rotating-credentials-for-github-com-and-new-ghes-patches/","impact":"minor","timestamp":"Dec 27, 02:51 - 03:06 UTC"},{"code":"d3mfyt4nqnv5","name":"Incident with Packages, Issues, Pull Requests and Webhooks","message":"On December 19, 2023, between 20:30 and 22:50 UTC multiple services experienced intermittent errors communicating with our MySQL backends. An average of 0.12% and max of 1.30% of requests hit 5xx errors and timeouts for web and API requests during this period. The majority of impact was mitigated at 22:10, with a lower rate of failures ending by 22:50. This issue was caused by unhandled exceptions during routine maintenance of our internal networking hosts. Hosts that were thought to be no longer serving traffic were removed from service which led to the initial errors. Recovery took longer than expected because of issues in our automated recovery tooling. As a result, manual intervention was performed to remove the unhealthy hosts and requests returned to normal. The bug in the automated recovery tooling has been fixed and we’ve made changes to prevent unintended removal of hosts that are serving traffic.","impact":"minor","timestamp":"Dec 19, 21:12 - 22:50 UTC"},{"code":"476y2pqmb3hc","name":"Incident with Git Operations and Pull Requests","message":"At 2023-12-18 20:03 UTC, our monitoring flagged an increase in GitRPC response errors that resulted from a change that was made that caused backend replicas to temporarily become out of sync while the change was deployed. During this time, approximately 2,500 users across 8,200 repos would have seen the merge button disabled or received failures from the merge API endpoint. Upon discovering the failures, our engineers assessed that the shortest path to resolution was to roll forward and manually deploy to the remaining hosts, which resolved the issue at 2023-12-18 21:26.","impact":"minor","timestamp":"Dec 18, 21:17 - 21:26 UTC"},{"code":"fj7v3z6fy7ym","name":"Incident with Copilot","message":"On December 1st, GitHub’s internal metrics highlighted increased latency for Copilot Chat requests. We declared an incident while investigating and discovered that the issue was only affecting GitHub employees. It was caused by a performance issue in a recently enabled feature that was only available internally.

We resolved the issue by disabling the feature.

The team has invested in improved telemetry, monitoring, and playbooks for Copilot Chat to ensure we can accurately assess impact and respond appropriately to alerting.","impact":"minor","timestamp":"Dec 1, 15:49 - 18:16 UTC"}]}],"show_component_filter":false,"show_uptime_calendar":true,"component_filter":null,"start_time":"2023-12-01T00:00:00Z","end_time":"2024-02-29T23:59:59Z"}

Jan	FEB	Mar
	23
2023	2024	2025

Incident History