We published a postmortem on our June 27th incident:
Follow
Python Package Index
@pypi
The Python Package Index (PyPI) is a repository of software for the Python programming language.
Follow for service updates, advisories, and curiosities.
Python Package Index’s Tweets
Resolved: This incident has been resolved.
1
2
3
Show this thread
Monitoring: We've disabled the specific tasks causing delays in the queue and are continuing to monitor the queue as it drains.
2
1
1
Show this thread
Identified: We've identified specific jobs in our queue that are causing processing delays and are working to remove them.
1
1
Show this thread
Investigating: A delay in our job queue is affecting CDN purges causing public pages to be stale and delays in email sending for notifications, email validation, password resets, etc.
1
2
5
Show this thread
We truly appreciate for demonstrating their support of PyPI as a Contributing sponsor. Their generosity ensures that we can improve and sustain PyPI for many generations to come!
5
19
Topics to follow
Sign up to get Tweets about the Topics you follow in your Home timeline.
Carousel
We've published an incident report for the JSON API redirect loop outage today: status.python.org/incidents/lgpr
We understand the frustration, but are excited that on the other side of this outage PyPI is more cachable, performant, and reliable; dropping ~25rps from our backends.
3
12
56
Resolved: This incident has been resolved.
1
1
5
Show this thread
Monitoring: All purges of JSON API documents have completed. Our backends are recovering from the added load of repopulating the entire cache. Any failed purges may result in latent redirect loops for specific projects or releases in …
1
1
4
Show this thread
Identified: The cache purge has cleared all but projects starting with the letter `p`. Our estimates failed to take into consideration the popularity of project names on PyPI starting with p 🙃
3
5
16
Show this thread
Identified: We have started a task which will iterate over all projects and purge the cache for each individually. This will keep the PyPI backends from being overloaded by a completely bare cache. This process will take some time to …
1
3
5
Show this thread
Identified: Some cached responses are causing redirect loops for endpoints on the JSON API. We are working to determine how to clear these cached values without impacting the overall health of PyPI.
1
7
18
Show this thread
We published an incident report on our June 2nd file hosting outage:
2
5
Show this thread
Monitoring: A fix has been implemented and we are monitoring the results.
1
1
1
Show this thread
Investigating: We are currently investigating this issue.
1
1
1
Show this thread
Incident report on malicious takeover of ctx package on PyPI has been published.
Read details, mitigation, analysis, and more at python-security.readthedocs.io/pypi-vuln/inde
8
155
253
Thank you to for financially supporting PyPI through ’s sponsorship program. Every sponsorship has an impact on PyPI’s sustainability and maintenance. Considering being a sponsor? Email sponsors@python.org
5
We are grateful for 's support of PyPI as a Maintaining sponsor of . Sponsorship funds not only help us maintain what we have but will also help us with future improvements!
2
Monitoring: A fix has been implemented and we are monitoring the results.
1
1
Show this thread
Investigating: We are currently investigating this issue.
1
1
6
Show this thread
Thank you to for financially supporting PyPI through ’s sponsorship program. Every sponsorship has an impact on PyPI’s sustainability and maintenance. Considering being a sponsor? Email sponsors@python.org
1
14
Support from organizations like help keep PyPI running and allow us to continually keep improving it. Thank you!
2
9
Huge thank you to for sponsoring PyPI. Does your company rely on PyPI? ’s sponsorship program aims to build a more sustainable PyPI for the whole community and fund improvements to the entire packaging ecosystem python.org/sponsors/appli
7
10
Monitoring: Applications are coming back online and we are monitoring for stability.
1
1
5
Show this thread
Investigating: All backend services for PyPI are down due to a cascading failure in our deployment tooling. We are investigating and working on restoring service.
1
4
15
Show this thread
Thank you for supporting PyPI. Through sponsorships and grants, raised over $300,000 for PyPI’s use. Let’s keep that momentum going! python.org/sponsors/appli
1
6
We are looking for PyPI users who can participate in user interviews as we develop new features. More details at
1
47
48
We are grateful for 's support of PyPI as a Contributing sponsor of . Sponsorship funds not only help us maintain what we have but will also help us with future improvements!
3
8
46
Thank you for supporting PyPI. Because of your support, can continue to fund packaging improvements, benefiting millions of Python users around the world!
1
9
We truly appreciate for demonstrating their support of PyPI by becoming a Visionary sponsor. Their generosity ensures that we can improve and sustain PyPI for many generations to come!
13
It means so much to have organizations like financially support PyPI. Their generosity helps us maintain and support the tools that the Python community uses daily!
10
We truly appreciate for demonstrating their support of PyPI by becoming a Supporting sponsor. Their generosity ensures that we can improve and sustain PyPI for many generations to come!
1
31
We are grateful to for showing their support of PyPI by becoming a Supporting sponsor of . Your tremendous support makes a huge impact in helping us achieve our vision for the Python programming language and the Python community.
2
16
Thank you for supporting PyPI. Through sponsorships and grants, raised over $300,000 for PyPI’s use. Let’s keep that momentum going! python.org/sponsors/appli
4
21
We're so grateful to for their support of PyPI as a Maintaining sponsor of ! Sponsors like you help us build a more sustainable PyPI for the whole community and fund improvements to the entire packaging ecosystem.
1
Quote Tweet
looks like @pypi's bandwidth just crossed 1.5PB/day and I'm honestly not sure if I should be proud or terrified
obligatory shoutout to @fastly!
Show this thread
1
17
84