What happened?

On 2024-03-26 around 23:30 UTC, the FedRAMP version of Automation Cloud - govcloud.uipath.us was not available, it returned errors to all users that attempted to access the site. This incident occurred due to a backend network infrastructure update that impacted the network traffic and caused a complete service disruption.

What went wrong?
Our team was rolling out a change to our internal networking to improve scalability. The change had been verified in a testing environment. Unfortunately, when applying the change an engineer accidentally executed the steps in the wrong order and this led to an outage of the services. During this outage, our telemetry also encountered a regression which led to late detection of the service disruption. After the issue was brought to our attention, the teams rolled back the changes and the services were restored around 2024-03-27 15:40 UTC.

How will we prevent this in the future?
After performing a detailed internal analysis of the issue, we will be introducing additional layers of improvements to our workflow processes and telemetry. This includes automating any manual steps and ensuring that the workflow runs from an end-to-end process without any manual input required. We will also be increasing our detection telemetry with additional layers of monitoring to avoid gaps in our telemetry. For any preconditional changes, we will ensure there is an additional layer of peer reviews in place prior to rollout.