What happened:
On October 1st, 2024, around 2:30AM PST, customers accessing the Public Sector Orchestrator service lost the ability to access authenticated pages. Users would have experienced timeouts and pages failing to load when attempting to use the service via authentication. The duration of the issue was on October 1st, 2024, between the hours of 2:30AM to 5:30AM PST.

‌

What went wrong:
A planned upgrade/rollout for the Orchestrator service was pushed out to enable a newer authentication feature. The authentication networking configuration in the Public Sector Cloud encountered an incompatibility with the new authentication feature and this caused the existing authentication to break for users.

‌

How do we prevent this in the future?
We have since then reviewed the issues in our testing environments to isolate the issues that led up to the outage. We have worked with our internal teams to introduce some additional testing enhancements around the authentication model to identify such issues prior to a production rollout. We have since then applied the fixes to the configuration in our test environments and have applied and updated configuration to the Public Sector Cloud as well. With these additional improvements and changes to our proactive testing, we will be able to identify issues within our testing environment ahead of time so it does not impact our production rollouts.