Refreshed
Welcome to the Turn.io status page. We will update any changes to system performance here. Subscribe to updates to receive email notifications.
Started:
Ended:
Length:
Started:
Ended:
Length:
Started:
Ended:
Length:
Started:
Ended:
Length:
Started:
Ended:
Length:
Resolved | Mar 21, 2023 | 13:31 GMT+01:00
This issue has been resolved.
Monitoring | Mar 21, 2023 | 13:11 GMT+01:00
The issue has been resolved. An underlying piece of hardware had failed and all impact services were automatically migrated and restarted on a new server instance. Apologies for the inconvenience.
Open | Mar 21, 2023 | 12:49 GMT+01:00
We're noticing that services on our Turn SaaS services are being restarted by our cloud hosting environment. We are busy investigating what the underlying cause is.
This is causing message delivery delays and user interface connection errors.
Resolved | Feb 23, 2023 | 14:40 GMT+01:00
Issue is resolved. Collection updates again as expected every 5mins.
Open | Feb 22, 2023 | 15:11 GMT+01:00
We have identified an issue with Collection counts not updating. This means that the chat/message list in the Conversation view may have more or less chats/messages than you are seeing on the Dashboard view.
The above issue is impacting all customers, both on the shared cluster and on private clouds.
We are working to resolve the issue as quickly as possible.
Resolved | Feb 22, 2023 | 15:06 GMT+01:00
Duplicate reminders issue has been resolved.
Monitoring | Feb 06, 2023 | 12:37 GMT+01:00
We have shipped a fix for the duplicate message delivery observed in Reminders, and will continue to monitor it closely. The feature can be used again as usual.
Open | Feb 03, 2023 | 08:02 GMT+01:00
We are experiencing issues with Reminders resulting in multiple messages being sent to contacts. This could impact quality ratings and we therefore recommend to not use Reminders until the issue is resolved.
The above issue is impacting all customers on our shared cluster at as well as private cloud customers.
We are actively working to resolve it.
Resolved | Jan 06, 2023 | 12:01 GMT+01:00
The issue related to stacks with buttons and list is resolved.
Open | Jan 06, 2023 | 10:56 GMT+01:00
We have identified an issue with Stacks. The stacks that have button cards and emojis in the list items either fails to send an intended response or no response at all.
The above issue is impacting all customers on our shared cluster at https://whatsapp.turn.io.
We are working to resolve the issue as quickly as possible.
Resolved | Oct 25, 2022 | 13:43 GMT+02:00
WhatsApp reported message sending and receiving has been restored.
Open | Oct 25, 2022 | 10:21 GMT+02:00
WhatsApp reported major disruptions impacting the ability to send messages via the WhatsApp Business Platform. Their teams are working to mitigate the issue.
Resolved | Oct 14, 2022 | 17:14 GMT+02:00
This issue has been resolved, the backlog on the low priority queue has been cleared.
Open | Oct 14, 2022 | 17:02 GMT+02:00
Turn operates 3 levels of queue priority, low, default, and high. The low priority queue is drained with least urgency and the high priority queue is given maximum amount of resources to ensure fast delivery.
Which queues your events are dispatched to depends on how fast your endpoint responds. The response time of your endpoints are sampled once every 20 webhook event deliveries. The time it takes you to accept a message for delivery determines what queue the following event is going to be dispatched to. Please read https://whatsapp.turn.io/docs/api/settings#application-settings for the details and the response time based prioritisation.
For the past few hours there has been a steady build-up on our low-priority webhook delivery queue due to clients' webhooks failing to be delivered as a result of an AWS outage in India. We've allocated extra resources to the low priority queue to get its backlog to clear.
The default and high priority queues are not seeing any queue build up.
We will update here when it's completed.
Resolved | Aug 12, 2022 | 14:40 GMT+02:00
The issue has been resolved and media files work as expected again.
Open | Aug 12, 2022 | 14:26 GMT+02:00
We have identified an issue with sending and receiving media files.
The above issue is impacting all customers on our shared cluster at https://whatsapp.turn.io. Private cloud customers are not affected.
We are working to resolve the issue as quickly as possible.
Resolved | Aug 11, 2022 | 14:15 GMT+02:00
No further message delivery delays have been observed, this issue is resolved.
Monitoring | Aug 11, 2022 | 12:49 GMT+02:00
The rate limiting issue has been addressed and messages are now being delivered. We are continuing to monitor the situation.
Open | Aug 11, 2022 | 12:45 GMT+02:00
We have identified an issue with rate limiting on the Turn API affecting message delivery.
The above issue is impacting all customers on our shared cluster at https://whatsapp.turn.io. Private cloud customers are not affected.
We are working to resolve the issue as quickly as possible.
Resolved | Jul 28, 2022 | 11:37 GMT+02:00
The issue has been resolved.
Open | Jul 28, 2022 | 11:01 GMT+02:00
We have identified an issue with multiple automated responses being send. We are working to resolve the issue as quickly as possible.
The above issue is impacting all customers on the shared Turn cluster at https://whatsapp.turn.io. Private cloud customers are not affected.
Resolved | Jul 12, 2022 | 14:16 GMT+02:00
We have resolved the issue that was causing delay in appearance of messages in chat list and search results.
Open | Jul 12, 2022 | 13:12 GMT+02:00
We have identified an issue due to which the messages appear in the chat list and in the search result with a delay. This is impacting all clients.
We are working to resolve the issue.
Resolved | Jul 02, 2022 | 05:23 GMT+02:00
The issue around automation is resolved for all clients.
Open | Jul 01, 2022 | 08:03 GMT+02:00
We have identified an issue with automations. Adding a new automation from turn interface returns a black screen. This is impacting all clients including the clients on private clusters.
We are working to resolve it.
Resolved | Jun 14, 2022 | 17:13 GMT+02:00
The issue related to media messages delivery is resolved for all numbers. All systems are working as expected.
Open | Jun 14, 2022 | 16:48 GMT+02:00
We have observed that the media messages are not sent or received from all numbers on Turn shared cluster. We are investigating the issue. We hope to resolve it shortly.
Resolved | Jun 14, 2022 | 15:57 GMT+02:00
The issue related to message delivery is resolved fully.
Monitoring | Jun 14, 2022 | 11:45 GMT+02:00
The issue related to message delivery is resolved and we are monitoring our systems.
Resolved | Jun 04, 2022 | 09:27 GMT+02:00
The cause for message processing delay has been identified and rectified. This incident did not affect all numbers, however those numbers affected would have seen intermittent delays in message processing. Normal operation has been restored.
Investigating | Jun 04, 2022 | 06:36 GMT+02:00
We have identified further delays in message processing, Turn engineers are investigating.
Resolved | Jun 04, 2022 | 05:08 GMT+02:00
Whilst some additional message processing delay was observed whilst monitoring, no delays are currently being observed.
Monitoring | Jun 04, 2022 | 03:01 GMT+02:00
The cause for message processing delay has been identified and rectified. We are continuing to monitor the situation.
Open | Jun 04, 2022 | 02:15 GMT+02:00
There's currently a delay in messages being processed, Turn engineers are investigating.
Resolved | May 25, 2022 | 22:07 GMT+02:00
No further message processing delay has been observed, the issues has been resolved.
Monitoring | May 25, 2022 | 20:18 GMT+02:00
The cause for message processing delay has been identified and rectified. Only a select number of numbers were affected. We are continuing to monitor the situation.
Open | May 25, 2022 | 19:45 GMT+02:00
There's currently a delay in messages being processed, Turn engineers are investigating.
Resolved | May 03, 2022 | 19:06 GMT+02:00
The maintenance window has closed and all messages queued have been processed.
Open | May 03, 2022 | 18:11 GMT+02:00
There's currently a delay in messages being processed for search results due to an active maintenance window. We expect this to be resolved within an hour. All messages are queued and will be processed once the maintenance is done.
Your services are online and function, there's only a delay in some messages being visible in the chat list or the search results page.
Resolved | Apr 25, 2022 | 12:51 GMT+02:00
The new version has been released successfully.
Open | Apr 21, 2022 | 16:01 GMT+02:00
We are pausing message indexing for a few minutes as part of our current release. During this time there will be a delay for messages to appear in the Turn UI. Message sending and receiving will not be impacted.
This will affect customers on app.turn.io around 16:00 GMT+2. Private Cloud customers will not be impacted.
Resolved | Apr 19, 2022 | 11:50 GMT+02:00
The new version has been released successfully.
Open | Apr 19, 2022 | 11:20 GMT+02:00
Customers on app.turn.io might experience brief downtime at around 11:20am GMT+2 as a result of our current release. Private Cloud customers will not be impacted.
Resolved | Mar 30, 2022 | 06:50 GMT+02:00
The issue related to the message delivery is resolved for all clients.
Open | Mar 24, 2022 | 07:06 GMT+01:00
There is an issue identified by WhatsApp team at their own end that is impacting all clients on WhatsApp business platform.
Businesses running v2.39.1 and v2.39.2 of the On-Premises API are unable to respond to a customer-initiated conversation more than 7 days after the conversation was initiated. WhatsApp team is currently investigating the root cause and proposing a mitigation.
We are monitoring the communications from WhatsApp team.
Resolved | Mar 24, 2022 | 06:27 GMT+01:00
The issue around thread is now resolved.
Open | Mar 24, 2022 | 06:27 GMT+01:00
The issue around threads is now resolved.
Open | Mar 14, 2022 | 11:53 GMT+01:00
We have identified an issue with the threads with reply buttons. The threads fails to load or the questions cards inside the threads are missing. In some cases the threads fails to save.
The above issue is impacting few clients on our shared cluster at https://whatsapp.turn.io
We are working to resolve the issue.
Resolved | Mar 11, 2022 | 02:30 GMT+01:00
Further monitoring reveals no further delays in message receipt and delivery. This issue has therefore been resolved.
Monitoring | Mar 11, 2022 | 01:39 GMT+01:00
The cause of the delay has been identified and rectified. All numbers should be operational again. Turn engineers are continuing to monitor the situation.
Open | Mar 11, 2022 | 01:04 GMT+01:00
A delay in message receipt and delivery has been observed on certain numbers. Turn engineers are investigating.
Resolved | Feb 22, 2022 | 13:03 GMT+01:00
The issue related to message delivery is resolved for all clients.
Monitoring | Feb 22, 2022 | 13:00 GMT+01:00
On 19 February 2022 from 11:15 to 16:45 UTC and again on 21 February 2022 from 08:15 to 12:45 UTC message delivery was disrupted for a limited set of WhatsApp numbers hosted by Turn. During this time affected numbers would be unresponsive or slow to receive and process messages.
What caused it?
We will use a simple comparison to explain the incident for our non-technical users, and then elaborate with all the technical details.
Let's say we have a bus with WhatsApp numbers on Turn as passengers on the bus, and each passenger brought a lot of luggage with them. The luggage represents each service starting up to be able to send and receive messages. The bus broke down and another bus was called in immediately to transfer the passengers to, but the passengers had so much luggage that it took some time to do the transfer. During the time it took to do the transfer service interruption was experienced.Now let’s explain it in detail.
Turn hosts WhatsApp numbers on a collection of clustered virtual machines. These virtual machines need to be replaced as part of general maintenance or upgraded from time to time. The process of replacing virtual machines involves creating new virtual machines followed by moving workloads from the existing machines to the new machines. Moving workloads involves first stopping a workload on an existing virtual machine, then starting it again on a new machine.
In the general case this process of moving a workload from one virtual machine to another should not result in noticeable service disruption as stopping and starting of any given workload should complete within 30 seconds. However, during startup some but not all workloads can cause a spike in CPU usage. This can lead to CPU contention and a delay in process readiness if a large number of workloads exhibiting this behavior are started at the same time.
The incident on 19 February was triggered by an automated process replacing a number of virtual machines in Turn's hosting environment. However in this case the workloads on the virtual machines being replaced were inadequately distributed on the virtual machines being replaced in such a way that the majority of them would exhibit a large spike in CPU usage during startup. This led to CPU contention as the virtual machines were replaced and a large number of expensive processes restarted simultaneously, leading to significant delays in workloads being ready to process messages.
The incident on 21 February was caused in a similar fashion by CPU contention, however in this case it was triggered by rollout of configuration changes to the Turn hosting environment meant to address some of the problems encountered on the 19th. Specifically, workloads were redistributed more equally across all virtual machines in the hosting environment. However, even with more balanced workloads, CPU contention was still observed on some virtual machines during process startup, again resulting in significant delays in workloads being ready to process messages.
What are we doing about it?
To address this incident the following fixes having been released:
- Paused the automatic GCP upgrade schedule until we reduce the process startup times. Automatic GCP maintenance can still happen and cause disruption, so our key focus is on reducing process startup times.
- Rebalance workloads over virtual machines to reduce potential for CPU contention during workload startup.
Work is still ongoing on the following fixes:
- Optimize workloads to eliminate CPU usage spikes during startup and thus the potential for CPU contention.
- Reconfigure Turn hosting environment to be more conservative in terms of allowing workload disruption. This means that in the case of virtual machines being replaced that only a small number of workloads will be moved at a time to ensure that even if CPU spikes were to occur during startup, that they would not lead to CPU contention.
- Clearly schedule maintenance windows on Turn’s status page to surface potential for disruption.
Resolved | Feb 03, 2022 | 09:06 GMT+01:00
Upgrades are successfully done. All systems are working as expected.
Resolution in progress | Feb 02, 2022 | 14:30 GMT+01:00
The automatic upgrades of underlying infrastructure is in progress. This is the reason why messages are failing to deliver. We are monitoring and working to resolve the issue.
Open | Feb 02, 2022 | 12:58 GMT+01:00
We have observed some issues with messages delivery and receipt. This is impacting few clients on app.turn.io. We are working to resolve the issue.
Resolved | Jan 27, 2022 | 14:17 GMT+01:00
The issue around the automation page in Sandbox accounts is resolved for all accounts.
Open | Jan 26, 2022 | 15:03 GMT+01:00
We have identified that the automation page in the sandbox accounts is not loading.
This impacts all sandbox accounts only.
We are working to resolve the issue as soon as possible.
Resolved | Nov 16, 2021 | 19:48 GMT+01:00
Turn application and messaging service operation has now been restored.
Monitoring | Nov 16, 2021 | 19:20 GMT+01:00
Turn application and messaging services are now accessible but might still have degraded performance.
We are continuing to monitor the situation as Google continues to address a Cloud Networking outage.
Open | Nov 16, 2021 | 19:05 GMT+01:00
The load balancing service used to serve the Turn application and messaging services is currently experiencing a major outage. As a result Turn is unable to process messages at this time.
We are continuing to monitor the situation.
Resolved | Nov 05, 2021 | 13:07 GMT+01:00
Upgrades are completed. All systems are working as expected.
Open | Nov 05, 2021 | 11:44 GMT+01:00
The underlying hosting infrastructure is undergoing an automated upgrade. All of the servers behind https://app.turn.io is being upgraded sequentially, services may experience message delivery delays or login issues while the upgrade is applied.
Resolved | Nov 02, 2021 | 11:41 GMT+01:00
The upgrades are completed. All systems are working as expected.
Open | Oct 28, 2021 | 10:11 GMT+02:00
The underlying hosting infrastructure is undergoing an automated upgrade. All of the servers behind https://app.turn.io is being upgraded sequentially, services may experience message delivery delays or login issues while the upgrade is applied.
Resolved | Oct 21, 2021 | 09:59 GMT+02:00
The issue has been resolved.
Open | Oct 21, 2021 | 09:15 GMT+02:00
We have observed an issue with the Turn user interface not loading correctly or loading slowly. This is impacting some of our clients. Our team is working on the issue. We hope to resolve it soon.
Resolved | Oct 08, 2021 | 23:55 GMT+02:00
The upgrades have completed.
Open | Oct 08, 2021 | 22:40 GMT+02:00
The underlying hosting infrastructure is undergoing an automated upgrade. Each of the servers behind https://app.turn.io is being upgraded sequentially, services may experience message delivery delays while the upgrade is applied.
Resolved | Oct 05, 2021 | 06:35 GMT+02:00
This has been resolved by WhatsApp.
Open | Oct 04, 2021 | 18:39 GMT+02:00
Due to the global WhatsApp network being offline, Turn is unable to process messages.
Resolved | Oct 04, 2021 | 18:38 GMT+02:00
This has been resolved.
Monitoring | Sep 20, 2021 | 06:04 GMT+02:00
Message delivery delays have been resolved. The underlying cause was a queue server running out of disk space. We're continuing to attend to this problem to prevent a repeat.
Open | Sep 20, 2021 | 05:42 GMT+02:00
We're seeing a message queue buildup and some delivery delays on app.turn.io. We are attending to the issue and will provide updates here as we work towards fixing the problem.
Resolved | Sep 16, 2021 | 22:02 GMT+02:00
The optimisations have been applied, any pending messages have been processed.
Open | Sep 16, 2021 | 21:44 GMT+02:00
We are applying some database and infrastructure optimisations, these are causing some message delivery delays. All pending messages will be processed within a few minutes.
Resolved | Sep 14, 2021 | 18:58 GMT+02:00
This issue has been resolved as part of our latest deploy.
Open | Sep 08, 2021 | 15:59 GMT+02:00
We're experiencing an issue with pagination when you view a conversation from the Search page. Currently the new messages overwrites the first ones instead of getting appended which is why some of the messages are not visible in the Turn interface.
No messages are lost due to above issue. We are working on a fix. We hope to resolve it as soon as possible.
Resolved | Sep 14, 2021 | 18:15 GMT+02:00
This issue has been resolved. It was due to the deploy of a new release taking significantly longer than expected.
Open | Sep 14, 2021 | 15:54 GMT+02:00
We are experiencing an issue with automation on Turn and are working to get it resolved as quickly as possible. We'll provide updates here as we continue to investigate.
Resolved | Jul 22, 2021 | 12:54 GMT+02:00
We have resolved the issue of button automation. All features are working as expected for all clients.
Identified | Jul 22, 2021 | 10:40 GMT+02:00
We have identified an issue with the button automation. When users click on the button inside message templates that they have received then they either get a catch all message or no message at all from the service.
This is impacting all clients. We are working hard to get it resolved.
Resolved | Jul 04, 2021 | 00:02 GMT+02:00
The upgrades have completed and everything's been operating normally since 19:20 UTC.
Open | Jul 03, 2021 | 20:49 GMT+02:00
Due to an unannounced service upgrade, the cluster powering app.turn.io was restarted. This may cause some intermittent interruptions in messaging and delays in messages being available in search.
We'll update this status update once completed.
Resolved | Jun 25, 2021 | 14:08 GMT+02:00
The issue related to threads is now resolved. All features are working as expected.
Open | Jun 25, 2021 | 10:58 GMT+02:00
We have found an issue that is affecting threads feature. This is impacting some of the clients on app.turn.io. Issues such as saving threads and updating threads is associated with it. We are working tirelessly to get it resolved as quickly as possible.
Resolved | Jun 21, 2021 | 12:00 GMT+02:00
We have resolved the issue related to threads page. Threads page is working as expected for all clients.
Open | Jun 21, 2021 | 09:54 GMT+02:00
We have identified an issue with the Threads automation page. Threads page is giving a white screen and is failing to load. This is impacting all clients.
Our teams are working on the issue. We hope to resolve it soon.
Resolved | May 13, 2021 | 19:13 GMT+02:00
The issue of sending media messages and templated messages with button is now resolved. All systems are working as expected.
Monitoring | May 13, 2021 | 12:10 GMT+02:00
The issue persists at WhatsApp's end. Messages templates with Buttons and Media messages are impacted due to this outage. We are closely following the updates from WhatsApp.
Resolved | Apr 01, 2021 | 13:32 GMT+02:00
The issue pertaining to loading of conversations is now resolved. All features are working as expected.
Open | Apr 01, 2021 | 12:51 GMT+02:00
We have encountered an error at our end due to which either turn is not loading or the collections/automations are not available. This is impacting all clients on app.turn.io. It is not impacting private clouds.
Our engineers are working to get it resolved as soon as possible.
Resolved | Mar 22, 2021 | 02:21 GMT+01:00
Content distribution network outage ended. All systems are working as expected.
Open | Mar 19, 2021 | 09:04 GMT+01:00
The content distribution network that is used to serve the Turn frontend application is currently experiencing a major outage. This outage might impact some of our clients.
Some of our clients might not be able to load the turn application. We are monitoring the situation.
Resolved | Mar 16, 2021 | 12:09 GMT+01:00
Clients on Infobip can now send and receive messages. The issue is resolved now.
Open | Mar 16, 2021 | 08:33 GMT+01:00
We have observed that the message delivery for clients on Infobip BSP fails. This is happening for all inbound and outbound messages. The issue is impacting only those clients who are on Infobip.
We are working to resolve the issue as soon as possible.
Resolved | Mar 04, 2021 | 06:39 GMT+01:00
We have upgraded our search infrastructure and message history is now available for 12 months.
Resolution in progress | Feb 22, 2021 | 07:08 GMT+01:00
We have restored the chat history for past 12 months for all app.turn.io clients. The backfill for private cloud clients are now in progress and should be completed in the next 2 days. We have also upgraded our search infrastructure. This means that the time taken for execution of search query is reduced. This also means that the performance of loading conversations, receiving and sending messages is also improved.
Resolution in progress | Feb 17, 2021 | 12:38 GMT+01:00
We have started the process to restore your full message history. As a result, search results and totals shown in Collections may vary more than expected. We will update you once the history is fully restored.
Resolution in progress | Feb 11, 2021 | 09:11 GMT+01:00
We're performing a system upgrade in response to performance issues with our search infrastructure. This will temporarily limit Search history to active users to the last 6 weeks for all our clients. As a result, the totals shown in Collections will only reflect users who have interacted with your service since 18 December 2020. No other features will be impacted except for Collections and Search.
To can learn more visit:
https://learn.turn.io/product-updates/10-february-2021-temporary-search-limitations-while-we-do-a-search-feature-upgrade
Resolved | Feb 24, 2021 | 16:08 GMT+01:00
All automations are now available.
Open | Feb 23, 2021 | 16:30 GMT+01:00
We have found an issue in our system which is causing availability of only 50 automation rules. All the rules that are not available are working fine for the users but those rules are not visible.
This is impacting all clients including clients on private clouds.
We are working to get it resolved soon.
Resolved | Feb 04, 2021 | 11:06 GMT+01:00
We have deployed a fix for this issue. The changes that we have done in our infrastructure have not only solved the loading conversations/chat issue but also reduce the time of loading chats.
Chats/conversations should now be working properly for all clients including clients on private clouds.
Resolved | Jan 29, 2021 | 11:00 GMT+01:00
Conversation are loading for all clients now.
Monitoring | Jan 26, 2021 | 11:47 GMT+01:00
We have applied a fix for the issues of loading conversation and display of chats on dashboard. All clients can now have a smooth loading of their conversations and should now have all the chats displayed in their dashboards.
We are, however, keeping this issue under monitoring.
Open | Jan 25, 2021 | 14:27 GMT+01:00
The issues of loading conversation and availability of chats in dashboard are now impacting all clients including clients on private clouds.
Our engineering teams are constantly working on a fix.
Resolution in progress | Jan 25, 2021 | 12:49 GMT+01:00
We are making progress on the issue of loading conversations. Currently, conversations are loading fine for all clients.
There is however a related issue of availability of new chats in the dashboard. New Chats are not reflected in the client's dashboard. This is affecting all clients on app.turn.io.
We are as eager as you are to solve the issue and we are working tirelessly to achieve that.
Open | Jan 25, 2021 | 09:46 GMT+01:00
Issue of loading collections still persists for all the clients on app.turn.io. We are working on it to get it resolved.
Resolution in progress | Jan 22, 2021 | 08:54 GMT+01:00
We have applied some changes to improve the performance. We are keeping it under watch and working on a fix.
Open | Jan 20, 2021 | 07:01 GMT+01:00
Conversations fails to load currently. This issue is affecting most of the clients on app.turn.io . This issue is not affecting the clients with private clouds. We are working hard to resolve it.
Resolution in progress | Jan 19, 2021 | 06:28 GMT+01:00
Collections are now loading but we are experiencing a degraded performance. It is taking more time than usual to load collection. We are working hard to get it resolved.
Monitoring | Jan 18, 2021 | 12:48 GMT+01:00
We have corrected our systems and now the conversations are loading properly. We are monitoring our system for next few hours.
Open | Jan 18, 2021 | 08:01 GMT+01:00
We have found an issue with app.turn.io which is primarily affecting loading of conversations in most of the whatsapp Bot lines.
We are working hard to get it resolved as quickly as possible.
Resolved | Jan 05, 2021 | 17:24 GMT+01:00
We've been observing this issue over the holidays and it is now resolved.
Open | Dec 23, 2020 | 12:32 GMT+01:00
If you continue to have problems logging and get an error about an "invalid token" please delete the browser cookies for https://app.turn.ioa and https://whatsapp.turn.io and refresh
Open | Dec 23, 2020 | 11:06 GMT+01:00
We've applied some extra security policy's to Turn. If you are finding yourself unable to connect, or disconnected, you need to logout and log back in again.
Resolved | Dec 23, 2020 | 12:50 GMT+01:00
This issue is now resolved.
We've identified the root cause to relate to big query synchronisation for a number of clients experiencing above average messaging throughputs. We've disabled the big query synchronisation for one account and are working on a permanent fix.
Open | Dec 23, 2020 | 12:31 GMT+01:00
We're experiencing some interruptions with message delivery. We are investigating the root cause.
Resolved | Dec 22, 2020 | 11:22 GMT+01:00
The issue was resolved, all backlogged messages that were still waiting to be made available in the search and collections results have been processed.
Open | Dec 21, 2020 | 14:12 GMT+01:00
We're seeing a backlog build up on our search infrastructure. This can result in a delay in messages and chats becoming visible in collections. We're working on resolving this.
Messaging is unaffected by this.
Resolved | Dec 18, 2020 | 15:56 GMT+01:00
The issue has been resolved.
Investigating | Dec 18, 2020 | 11:09 GMT+01:00
While messages are being processed as normal, they may not be showing up in search results or in the chat list yet.
We are continuing to investigate the issues with the search infrastructure and will update here as soon as we know more.
Open | Dec 18, 2020 | 10:26 GMT+01:00
We're experiencing down time due to problems with indexing of messages on our search infrastructure.
We're investigation and will update as soon as we know more.
Resolved | Dec 14, 2020 | 15:21 GMT+01:00
Google seems to have resolved most of their issues.
Open | Dec 14, 2020 | 13:13 GMT+01:00
Google is having a global outage and Turn relies on Google for login and account management.
While Google is having these problems, you may face issues while logging into any Turn instance.
Resolved | Nov 27, 2020 | 15:23 GMT+01:00
The issue has been resolved.
Open | Nov 27, 2020 | 11:42 GMT+01:00
We've found an issue with automations that is impacting numbers who have more than 50 pieces of content loaded in Turn. We're busy addressing this issue.
Resolved | Nov 10, 2020 | 16:42 GMT+01:00
The issue has been resolved for all Turn installations.
Monitoring | Nov 10, 2020 | 13:11 GMT+01:00
The issue is resolved on app.turn.io, we're monitoring the situation on the private cloud installations.
Resolved | Oct 21, 2020 | 14:13 GMT+02:00
The issue has been resolved for the 2 clusters impacted. The underlying technical root cause was a database migration causing a table lock on one of the applications' main tables. This was further complicated by automated database cleanup processes happening on the same table a the same time.
Our apologies for the downtime experienced.
Investigating | Oct 21, 2020 | 12:44 GMT+02:00
The fix we've applied has not yet had it's intended effect. We're continuing to investigate.
Resolution in progress | Oct 21, 2020 | 11:49 GMT+02:00
A fix is being released to the production environments impacted.
Open | Oct 21, 2020 | 11:17 GMT+02:00
Unfortunately we pushed a faulty release to 2 of our production clusters. This is causing downtime for https://app.turn.io. We are busy issuing a fix and the systems should be online again shortly.
Our sincere apologies for this inconvenience, we will provide updates here as we address the underlying issue.
Resolved | Oct 08, 2020 | 14:07 GMT+02:00
The issue has been resolved.
Open | Oct 08, 2020 | 13:03 GMT+02:00
We've found an issue where Turn is not accepting status updates on messages. This can mean that you'll experience a delay in seeing your messages marked as read, sent, or delivered. We're busy addressing this issue.
Resolved | Sep 15, 2020 | 10:47 GMT+02:00
All system upgrades have been applied.
Open | Sep 15, 2020 | 10:14 GMT+02:00
We're performing a system upgrade in response to Google's https://cloud.google.com/kubernetes-engine/docs/security-bulletins#gcp-2020-012 security bulletin on all our Google Cloud Platform hosted systems. This will involve various restarts across the platforms. Any messages received during a restart will be handled immediately as the restarts of the various components are completed.
Resolved | Sep 15, 2020 | 10:07 GMT+02:00
We've observed this for a few days and it is now resolved.
Monitoring | Sep 11, 2020 | 13:38 GMT+02:00
We've put some more fixes in places and are observing the system's behaviour. This issue seems to be related to automatic upgrades performed by the hosting services that app.turn.io is using. This is not impacting any of the private cloud setups.
Investigating | Sep 11, 2020 | 12:01 GMT+02:00
The fix we put in place is no longer resolving this issue we are experiencing with database connections. We're investigating.
Resolved | Sep 11, 2020 | 10:43 GMT+02:00
The issue is resolved. We're continuing to monitor the systems.
Open | Sep 11, 2020 | 10:31 GMT+02:00
We've found an issue with a number of numbers unable to receive messages from WhatsApp. We've identified the issue and are busy addressing it.
Any messages currently pending delivery will be processed shortly.
Resolved | Aug 25, 2020 | 09:23 GMT+02:00
Collection search indexing has been optimised to reduce load on the search backend. Search query and collection performance has been greatly improved as a result.
We're continuing to work on improvements to prevent this happening in future - in particular we're changing the way All Conversations is architected so that it is much faster and doesn't rely on search
Monitoring | Aug 25, 2020 | 09:22 GMT+02:00
Collection search indexing has been optimised to reduce load on the search backend. Search query and collection performance has been greatly improved as a result.
We're continuing to work on ways to prevent this occurring in future including changing the way All Conversations is architected so that it is much faster and doesn't rely on search.
Resolution in progress | Aug 19, 2020 | 14:35 GMT+02:00
To alleviate some of the load we'll be temporarily be turning off the total user count display for collections.
Resolution in progress | Aug 19, 2020 | 13:39 GMT+02:00
Currently, Collections rely on Search functionality to find conversations that match the Collection criteria. The All Conversations Collection, in particular, puts a high load on Search, slowing down the performance across all other Collections.
We're busy working on a change to how the All Conversation Collection functions to ensure that is performs faster.
Investigating | Aug 18, 2020 | 12:42 GMT+02:00
Some users are experiencing issues where Collections are loading a blank screen, or partial screen.
We're exploring different reasons this may be happening.
Resolved | Aug 19, 2020 | 13:38 GMT+02:00
Content is now behaving as usual on app.turn.io
Resolution in progress | Aug 19, 2020 | 10:58 GMT+02:00
We found an issue where content is no longer editable due to a change we made in the underlying content format.
This has now been reverted. Please do a hard refresh to ensure you see the changes.
Resolved | Jul 31, 2020 | 17:25 GMT+02:00
A new release is being deployed to all the clusters, all inbound media for Infobip numbers will be working again as expected within a few minutes.
Resolution in progress | Jul 31, 2020 | 12:23 GMT+02:00
QA passed, we'll be releasing to all our production environments shortly.
Resolution in progress | Jul 31, 2020 | 10:55 GMT+02:00
We've identified the issue and have a fix in place. We're currently testing it in our QA environment.
Identified | Jul 31, 2020 | 10:28 GMT+02:00
This seems to be limited to numbers backed by Infobip. We're continuing to dig into this issue.
Open | Jul 31, 2020 | 09:03 GMT+02:00
We're seeing an issue where media received via WhatsApp is not displaying properly in the UI nor accessible via the API.
We are busy investigating and will provide updates as soon as possible.
Resolved | Jul 28, 2020 | 17:27 GMT+02:00
Issue is resolved
Closed | Jul 28, 2020 | 17:26 GMT+02:00
The issue has been resolved.
Resolution in progress | Jul 28, 2020 | 15:24 GMT+02:00
We've indentified one possible root cause which could result in processed messages being rescheduled for processing again by the automation system.
Resolution in progress | Jul 28, 2020 | 14:47 GMT+02:00
The backlog of background jobs has been processed and things should be snappy again. We're continuing to investigate the root cause of this issue.
Identified | Jul 28, 2020 | 14:37 GMT+02:00
We've identified a situation where a number is constantly rescheduling new jobs in the background job queue causing it to grow constantly.
Open | Jul 28, 2020 | 14:08 GMT+02:00
Due to a problematic release, there are quite a few messages pending to be processed and still due for delivery. We are working through the backlog and services relying on the automation system should be responsive again shortly.
Resolved | Jul 22, 2020 | 14:20 GMT+02:00
This issue is now resolved. Please get in touch with us if you are having issues with conversations or messages not being visible in the UI search results or collections.
Monitoring | Jul 22, 2020 | 13:41 GMT+02:00
Our search infrastructure upgrade has completed and we're processing a backlog of conversations and messages needing to be processed by the search backend. Once they've been processed they will show up in your search results and collections overviews. We're expecting this to be completed within the next hour.
Open | Jul 22, 2020 | 11:03 GMT+02:00
We've reprovisioned the search infrastructure and are re-indexing the chat histories for all clients. While this in progress clients may have an incomplete view of the chats in search results and collections.
Open | Jul 20, 2020 | 14:22 GMT+02:00
We're noticing we're hitting some infrastructure capacity limits on our app.turn.io hosting environment. We're busy adding more capacity to meet the demand.
Resolved | Jun 30, 2020 | 12:23 GMT+02:00
This issue has for now been resolved.
Monitoring | Jun 29, 2020 | 12:29 GMT+02:00
Services are operating as normal again. The search infrastructure is doing better but the underlying cause is not fully resolved yet. We're working on a more permanent solution.
Open | Jun 29, 2020 | 11:22 GMT+02:00
Our Elasticsearch installation is suffering from degraded performance. This is causing the UI to load slowly for many users. We are working on restoring service as soon as possible.
Messaging services should be operational as normal.
Resolved | Jun 10, 2020 | 12:06 GMT+02:00
This issue has been resolved.
Open | Jun 10, 2020 | 11:52 GMT+02:00
Due to a new software release, some messages are not being delivered to some countries.
A fix is being released and should be resolved shortly.
Resolved | May 30, 2020 | 13:26 GMT+02:00
We've updated how we route traffic to whatsapp.turn.io and the issue has been resolved by issuing a different SSL certificate.
Open | May 30, 2020 | 13:12 GMT+02:00
We have found an issue with an SSL certificate's authority certificate having expired which is causing some downtime for some clients.
We're working on resolving this.
Resolved | Feb 27, 2020 | 15:43 GMT+01:00
This issue has been resolved.
Resolution in progress | Feb 07, 2020 | 12:44 GMT+01:00
For clients with many collections spanning hundreds of thousands of monthly active users the Turn dashboard page was having trouble loading. We isolated the issue to collections taking longer to load than could reasonably be expected.
We are working on a fix to address this issue for these clients.
Resolved | Feb 26, 2020 | 12:21 GMT+01:00
This issue has been resolved.
Monitoring | Feb 24, 2020 | 13:29 GMT+01:00
We're investigating some issues reported with images not being delivered via Infobip.
Monitoring | Feb 12, 2020 | 08:48 GMT+01:00
We are continuing to monitor media attachment behaviour over the next few days.
Investigating | Jan 31, 2020 | 10:48 GMT+01:00
We have found an issue where attachments such as location pins and audio files are displaying incorrectly in the Turn UI.
The Turn team is working to get it resolved as quickly as possible.
Resolved | Feb 12, 2020 | 08:47 GMT+01:00
Faktory has been behaving as expected over the last 5 days and therefore this incident is considered resolved.
Monitoring | Feb 07, 2020 | 16:19 GMT+01:00
A temporary fix was put in place at 9:07 UTC. A permanent fix will be deployed shortly.
Monitoring | Feb 07, 2020 | 12:43 GMT+01:00
We have found an issue with Turn and are working tirelessly to get it resolved as quickly as possible.Turn off-loads the delivery of webhooks and indexing of messages and conversations to a background process. This allows Turn’s APIs and frontend to remain performant by delegating tasks that do not require immediate processing, or tasks which are allowed to take a little longer to complete, and to continue working in the background while Turn continues to focus on message processing.
This morning the piece of Turn infrastructure that is responsible for accepting and scheduling these tasks ran into its maximum allowance of disk space. As a result, it intermittently refused to accept more tasks as it briefly had no disk space to store them.
The impact on clients was that webhook deliveries were delayed and the search and conversation listings were at times showing outdated information.
We increased the disk space allocation for the background worker and we have reindexed all the messages and conversations for all clients to make sure the frontend is displaying the latest messages and updated conversations again.
We are putting measures in place to make sure the disk space allocations increase as our clients’ monthly active users and message volumes increase.