Resolved -
Everything has returned to normal and the job queue is back down close to zero.
Nov 19, 22:10 UTC
Update -
Update 3:02pm CT:
We are still working on the core issue, but we are beginning to resume background operations. Currently, we are running at partial capacity for enrichment, materialization, queue, and sync jobs.
Manually-initiated jobs (i.e. jobs started by clicking a button in the dashboard) will be prioritized in the queue.
Please expect longer than usual queue times on sync jobs.
Nov 18, 21:04 UTC
Identified -
We are currently experiencing a database pressure issue in our US Central region (which holds most of our data). In an abundance of caution, while we sort out the issue, we have voluntarily disabled some nonessential background services, including:
- Scheduled syncs
- Data enrichments
- Materializations
- Queued jobs & tasks
***Core platform functionality (API, SSO, Dashboard, Widgets, etc.) remains fully operational.***
What’s Happening:
An internal database counter exceeded its safe threshold. When this number gets too high, the database temporarily prevents new write operations. In order to slow the growth of this counter while we fix the issue, we've reduced the number of workloads on the database.
What We’ve Done So Far:
- Identified the cause
- Temporarily paused non-critical syncing and materialization tasks during recovery
- We are manually running certain database maintenance processes to help the database recover more quickly
Current Impact:
- Only background data tasks are delayed (intentionally)
- No data has been lost
- Primary application operations continue as normal
Next Steps:
We will monitor the configuration update. If it's enough, we'll bring non-essential services back online. If it isn't, we may need to restart the database which will happen at non-peak time and require < 10 minutes of downtime.
Next Update:
We will provide another update at the end of the day, or if the situation resolves or escalates, whichever occurs first.
Nov 18, 18:06 UTC