We have not observed a single 502 across our systems for the past two hours, while close to peak load.  We will consider this issue resolved as there is no current customer impact. 

We will continue to monitor and evaluate internally in order to prevent any future disruption.

Elevated 502 Errors

The incident has been confirmed as resolved.

Performance Degradation

At approximately 21:00 UTC on 5/2/22 - customers began to see intermittent 502s as well as slow down in response times.   These rates increased through the night and were resolved at approximately 13:39 on 5/3/22.

Root cause was determined to be an influx of requests from an outside source.  We have scaled the system accordingly and will continue to monitor the status closely.

Going forward we will continue to 1) investigate the root cause of the issue as well  and 2) investigate as to why our alerting system failed to notify the support team.

This is our first customer impacting outage in over 3 years and we'd like to sincerely apologize to any of our partners who were impacted.

502s and Slow Response Times

During a server upgrade, there was approximately 1-2 minutes of API downtime.   This was unintentional and we are looking into why this happened so we will avoid future outages in the future as we upgrade hardware.

API Outage

Customer Impact: Access to our storage APIs (publishing/reading packages) was returning 500 errors for some partners.

This is a repeat of the May 9th outage - please refer to the incident summary for more details.

Resolution: After we were alerted of this issue we were able to restore functionality to all partners.

Duration: Approximately 45 minutes at approximately 11:00 CST and 20 minutes around 18:00 CST.

Future Mitigation: To prevent this from happening again, we will be implementing several changes:

1) Improve our monitoring to detect 500 errors as soon as they occur. 
2) Increased the size of our cluster to give us more headroom in our connection pools.
3) Continue to investigate root cause and fix anything that may be holding on to connections.

Package Repository Outage

Customer Impact: Access to our storage APIs (publishing/reading packages) was returning 500 errors for some partners.

Root Cause:  Our servers exhausted their connections to the storage layer and our monitoring system did not alert us to this degraded state - a partner alerted us instead.

Resolution: After we were alerted of this issue we were able to restore functionality to all partners.

Duration: Approximately two hours

Future Mitigation:  To prevent this from happening again, we will be implementing several changes:

1) Improve our monitoring to detect 500 errors as soon as they occur.   
2) Increased the size of our cluster to give us more headroom in our connection pools.
3) Continue to investigate root cause and fix anything that may be holding on to connections.

Start Time (UTC)	Incident Deep Link	Impact	Duration	Description
2025-01-13 07:37	Elevated 502 Errors	minor	2 hrs, 50 mins	We have not observed a single 502 across our systems for the past two hours, while close to peak load. We will c...read more
2022-08-16 12:13	Performance Degradation	none	1 hr, 39 mins	The incident has been confirmed as resolved.The incident has been confirmed as resolved.
2022-05-02 16:00	502s and Slow Response Times	major	0 second	At approximately 21:00 UTC on 5/2/22 - customers began to see intermittent 502s as well as slow down in response ...read more
2018-06-06 09:34	API Outage	none	0 second	During a server upgrade, there was approximately 1-2 minutes of API downtime. This was unintentional and we are...read more
2018-05-10 12:00	Package Repository Outage	critical	0 second	Customer Impact: Access to our storage APIs (publishing/reading packages) was returning 500 errors for some partn...read more
2018-05-09 12:00	Package Repository Outage	critical	0 second	Customer Impact: Access to our storage APIs (publishing/reading packages) was returning 500 errors for some partn...read more

CloudRepo Status

Is CloudRepo down?

Past CloudRepo Incidents