Resolved -
This morning, we experienced an issue with the execution of custom transformations during the final materialization phase of our data pipeline. Custom transformations are utilized by approximately 22% of our integrations. Integrations without custom transformations were not impacted. Integrations that utilized "Edlink-defined" transformations (such as "Infer Roles") were not impacted.
Timeline of Events
- Our transformation execution relies on private NPM packages in our NPM organization. - Around 3:12am CT, the first custom transformation failed because it was unable to load packages from NPM. - The first client issues were reported around 8:30am CT, but it was not apparent that this was a widespread phenomenon until approximately 9:50am CT when it was escalated to P1. - We resolved the issue at approximately 10:10am CT, but due to latency, materializations did not resume until approximately 10:23am CT.
Root Cause
The root causes was related to an expired credit card. After a failed payment, NPM unexpectedly downgraded our account and we were no longer able to access our own packages. We re-upgraded the account and after some brief latency due to their billing system, we are able to access packages again.
Changes Going Forward
1. In response, we plan to introduce additional alerting to detect unusual changes in our materializations. 2. We are also seeking alternatives to NPM for holding our private packages for materialization execution. Our use of NPM was a technical requirement when we first implemented our custom transformation requirement, but that may no longer be the case.
May 29, 08:00 UTC