Turnio - Notice history

This issue has been resolved.

For clients with many collections spanning hundreds of thousands of monthly active users the Turn dashboard page was having trouble loading. We isolated the issue to collections taking longer to load than could reasonably be expected. We are working on a fix to address this issue for these clients.

Faktory has been behaving as expected over the last 5 days and therefore this incident is considered resolved.

A temporary fix was put in place at 9:07 UTC. A permanent fix will be deployed shortly.

We have found an issue with Turn and are working tirelessly to get it resolved as quickly as possible.Turn off-loads the delivery of webhooks and indexing of messages and conversations to a background process. This allows Turn’s APIs and frontend to remain performant by delegating tasks that do not require immediate processing, or tasks which are allowed to take a little longer to complete, and to continue working in the background while Turn continues to focus on message processing. This morning the piece of Turn infrastructure that is responsible for accepting and scheduling these tasks ran into its maximum allowance of disk space. As a result, it intermittently refused to accept more tasks as it briefly had no disk space to store them. The impact on clients was that webhook deliveries were delayed and the search and conversation listings were at times showing outdated information. We increased the disk space allocation for the background worker and we have reindexed all the messages and conversations for all clients to make sure the frontend is displaying the latest messages and updated conversations again. We are putting measures in place to make sure the disk space allocations increase as our clients’ monthly active users and message volumes increase.

This issue has been resolved.

We're investigating some issues reported with images not being delivered via Infobip.

We are continuing to monitor media attachment behaviour over the next few days.

We have found an issue where attachments such as location pins and audio files are displaying incorrectly in the Turn UI. The Turn team is working to get it resolved as quickly as possible.

We have applied a fix to it.

While monitoring capabilities is an ongoing concern, here are some immediate steps we are undertaking today: 1. We are improving our processes around communicating issues on Turn as they unfold. These updates will arrive via email and have Turn Status Update as a subject prefix. These updates will come from the email address support@turn.io. Please make sure your email address is correct and up to date, if there are any other people you want to be updated around technical concerns of Turn please do let us know what email addresses to include. 2. We will be introducing a system availability status page to help give insight into Turn's uptime and general platform availability. We will notify you of the URL as soon as it is available. 3. We are investigating automated means of proactively highlighting any core functionality issues that we can implement in Turn to surface these concerns sooner. We will update you on our progress on this shortly. It is our sincere apologies that this bug resulted in you not being able to use Turn as expected.

On the 22nd of January at 12:18 UTC Turn version 4.0.31 was released and automatically deployed. Unfortunately this version introduced a bug which impacted numbers using a direct WhatsApp integration and relying on webhooks for service delivery, such as your account. In Turn, webhooks can be configured to be notified of inbound messages, outbound messages, or both. The result of this bug was that inbound messages were routed through to webhooks configured for receiving outbound messages. Any webhooks configured to receive inbound messages did not receive these messages unless that webhook was also configured as a recipient of outbound messages. At 10:43 UTC on the 23rd of January we were notified of this bug and a fix was deployed at 12:16 UTC on the same day in Turn version 4.0.34 For the duration between when the bug was introduced and when it was resolved, the Turn application and UI was online and functioning. What was impacted was the routing of messages to webhooks configured as recipients of inbound messages.

Issue has been resolved.

While monitoring capabilities is an ongoing concern, here are some immediate steps we are undertaking today: 1. We are improving our processes around communicating issues on Turn as they unfold. These updates will arrive via email and have Turn Status Update as a subject prefix. These updates will come from the email address support@turn.io. Please make sure your email address is correct and up to date, if there are any other people you want to be updated around technical concerns of Turn please do let us know what email addresses to include. 2. We will be introducing a system availability status page to help give insight into Turn's uptime and general platform availability. We will notify you of the URL as soon as it is available. 3. We are investigating automated means of proactively highlighting any core functionality issues that we can implement in Turn to surface these concerns sooner. We will update you on our progress on this shortly. It is our sincere apologies that this bug resulted in you not being able to use Turn as expected.

On the 22nd of January at 12:18 UTC Turn version 4.0.31 was released and automatically deployed. Unfortunately this version introduced a bug which impacted numbers using a direct WhatsApp integration and relying on webhooks for service delivery, such as your account. In Turn, webhooks can be configured to be notified of inbound messages, outbound messages, or both. The result of this bug was that inbound messages were routed through to webhooks configured for receiving outbound messages. Any webhooks configured to receive inbound messages did not receive these messages unless that webhook was also configured as a recipient of outbound messages. At 10:43 UTC on the 23rd of January we were notified of this bug and a fix was deployed at 12:16 UTC on the same day in Turn version 4.0.34 For the duration between when the bug was introduced and when it was resolved, the Turn application and UI was online and functioning. What was impacted was the routing of messages to webhooks configured as recipients of inbound messages.