Monitoring -
Instructure is currently undergoing a data breach (separate from the one earlier this week) and the attackers have gained control of many Canvas instances. Users of these instances will not be able to sign in or access any content. Data syncs (either to or from Instructure) are presumed non-functional at the moment.
Resolved -
We've identified the root cause and the pipeline has resumed. We will publish a full post-mortem shortly.
May 4, 19:57 UTC
Update -
We've isolated the issue to our sync service that retrieves data from school data sources. We are re-enabling enrichment & materialization. We are going to revert to the previous stable version of the sync application and re-enable it in a limited capacity for monitoring.
May 4, 18:30 UTC
Monitoring -
Reports of degraded API & SSO performance started coming in around 12:30pm CT. After investigation our logs, it appears that the degraded performance started at exactly 12:07pm Central Time (about 27 minutes prior to first reports). As of 1:04pm Central Time, we have reallocated database resources and the immediate issue has been resolved. SSO & API requests have returned to expected levels. We are continuing to investigate the root cause of the issue before restoring normal pipeline services.
Please expect delays in sync and materialization queues while we investigate the issue.
May 4, 18:16 UTC