Turnio - Turn Status: Background job worker running out of disk space | 7 February 2020 – Incident details

Faktory has been behaving as expected over the last 5 days and therefore this incident is considered resolved.

A temporary fix was put in place at 9:07 UTC. A permanent fix will be deployed shortly.

We have found an issue with Turn and are working tirelessly to get it resolved as quickly as possible.Turn off-loads the delivery of webhooks and indexing of messages and conversations to a background process. This allows Turn’s APIs and frontend to remain performant by delegating tasks that do not require immediate processing, or tasks which are allowed to take a little longer to complete, and to continue working in the background while Turn continues to focus on message processing. This morning the piece of Turn infrastructure that is responsible for accepting and scheduling these tasks ran into its maximum allowance of disk space. As a result, it intermittently refused to accept more tasks as it briefly had no disk space to store them. The impact on clients was that webhook deliveries were delayed and the search and conversation listings were at times showing outdated information. We increased the disk space allocation for the background worker and we have reindexed all the messages and conversations for all clients to make sure the frontend is displaying the latest messages and updated conversations again. We are putting measures in place to make sure the disk space allocations increase as our clients’ monthly active users and message volumes increase.