Resolved | Nov 27, 2024 | 12:28 GMT+01:00
We have deployed a fix that ensures that webhooks are disabled and this change communicated, in order to avoid this sort of system overload again.
Monitoring | Nov 19, 2024 | 10:08 GMT+01:00
Messaging via webhooks experienced service interruptions yesterday due to a remote endpoint not accepting webhook deliveries at 10:08. Automated escalations alerted our support engineers to it immediately and they moved to resolve it immediately. The queue was drained and full service was restored for all customers relying on webhook deliveries for their service by 11:08.
We'll be introducing a circuit breaker pattern in our webhook delivery system to guard against these situations causing message delivery problems going forward.
Unfortunately, and contrary to our normal practice, this status update was posted late.