Resolved -
We've identified the root cause and the pipeline has resumed. We will publish a full post-mortem shortly.
May 4, 19:57 UTC
Update -
We've isolated the issue to our sync service that retrieves data from school data sources. We are re-enabling enrichment & materialization. We are going to revert to the previous stable version of the sync application and re-enable it in a limited capacity for monitoring.
May 4, 18:30 UTC
Monitoring -
Reports of degraded API & SSO performance started coming in around 12:30pm CT. After investigation our logs, it appears that the degraded performance started at exactly 12:07pm Central Time (about 27 minutes prior to first reports). As of 1:04pm Central Time, we have reallocated database resources and the immediate issue has been resolved. SSO & API requests have returned to expected levels. We are continuing to investigate the root cause of the issue before restoring normal pipeline services.
Please expect delays in sync and materialization queues while we investigate the issue.
May 4, 18:16 UTC