Turn Status Page logo

All systems operational

Refreshed

System metrics

Monitors

Web App - app.turn.io

Uptime 6 days 31 minutes 9 seconds

Operational

24 hours

100.000%

Week

99.905%

Month

99.899%

Year

99.987%

API - whatsapp.turn.io

Uptime 6 days 28 minutes 49 seconds

Operational

24 hours

100.000%

Week

99.908%

Month

99.901%

Year

99.937%

Incident history

Service disruption due to load balancer outage

Resolved | Nov 16, 2021 | 19:48 GMT+01:00

Turn application and messaging service operation has now been restored.

+Show history

  • Monitoring | Nov 16, 2021 | 19:20 GMT+01:00

    Turn application and messaging services are now accessible but might still have degraded performance.

    We are continuing to monitor the situation as Google continues to address a Cloud Networking outage.

  • Open | Nov 16, 2021 | 19:05 GMT+01:00

    The load balancing service used to serve the Turn application and messaging services is currently experiencing a major outage. As a result Turn is unable to process messages at this time.

    We are continuing to monitor the situation.

Automated Infrastructure upgrades underway

Resolved | Nov 05, 2021 | 13:07 GMT+01:00

Upgrades are completed. All systems are working as expected.

+Show history

  • Open | Nov 05, 2021 | 11:44 GMT+01:00

    The underlying hosting infrastructure is undergoing an automated upgrade. All of the servers behind https://app.turn.io is being upgraded sequentially, services may experience message delivery delays or login issues while the upgrade is applied.

Automated Infrastructure upgrades underway

Resolved | Nov 02, 2021 | 11:41 GMT+01:00

The upgrades are completed. All systems are working as expected.

+Show history

  • Open | Oct 28, 2021 | 10:11 GMT+02:00

    The underlying hosting infrastructure is undergoing an automated upgrade. All of the servers behind https://app.turn.io is being upgraded sequentially, services may experience message delivery delays or login issues while the upgrade is applied.

Application loading issues

Resolved | Oct 21, 2021 | 09:59 GMT+02:00

The issue has been resolved.

+Show history

  • Open | Oct 21, 2021 | 09:15 GMT+02:00

    We have observed an issue with the Turn user interface not loading correctly or loading slowly. This is impacting some of our clients. Our team is working on the issue. We hope to resolve it soon.

Automated Infrastructure upgrades underway

Resolved | Oct 08, 2021 | 23:55 GMT+02:00

The upgrades have completed.

+Show history

  • Open | Oct 08, 2021 | 22:40 GMT+02:00

    The underlying hosting infrastructure is undergoing an automated upgrade. Each of the servers behind https://app.turn.io is being upgraded sequentially, services may experience message delivery delays while the upgrade is applied.

WhatsApp network offline

Resolved | Oct 05, 2021 | 06:35 GMT+02:00

This has been resolved by WhatsApp.

+Show history

  • Open | Oct 04, 2021 | 18:39 GMT+02:00

    Due to the global WhatsApp network being offline, Turn is unable to process messages.

Message queueing

Resolved | Oct 04, 2021 | 18:38 GMT+02:00

This has been resolved.

+Show history

  • Monitoring | Sep 20, 2021 | 06:04 GMT+02:00

    Message delivery delays have been resolved. The underlying cause was a queue server running out of disk space. We're continuing to attend to this problem to prevent a repeat.

  • Open | Sep 20, 2021 | 05:42 GMT+02:00

    We're seeing a message queue buildup and some delivery delays on app.turn.io. We are attending to the issue and will provide updates here as we work towards fixing the problem.

Message delivery delays

Resolved | Sep 16, 2021 | 22:02 GMT+02:00

The optimisations have been applied, any pending messages have been processed.

+Show history

  • Open | Sep 16, 2021 | 21:44 GMT+02:00

    We are applying some database and infrastructure optimisations, these are causing some message delivery delays. All pending messages will be processed within a few minutes.

Pagination issue when viewing conversations from Search

Resolved | Sep 14, 2021 | 18:58 GMT+02:00

This issue has been resolved as part of our latest deploy.

+Show history

  • Open | Sep 08, 2021 | 15:59 GMT+02:00

    We're experiencing an issue with pagination when you view a conversation from the Search page. Currently the new messages overwrites the first ones instead of getting appended which is why some of the messages are not visible in the Turn interface.

    No messages are lost due to above issue. We are working on a fix. We hope to resolve it as soon as possible.

Automation not working

Resolved | Sep 14, 2021 | 18:15 GMT+02:00

This issue has been resolved. It was due to the deploy of a new release taking significantly longer than expected.

+Show history

  • Open | Sep 14, 2021 | 15:54 GMT+02:00

    We are experiencing an issue with automation on Turn and are working to get it resolved as quickly as possible. We'll provide updates here as we continue to investigate.

Button Automation not working

Resolved | Jul 22, 2021 | 12:54 GMT+02:00

We have resolved the issue of button automation. All features are working as expected for all clients.

+Show history

  • Identified | Jul 22, 2021 | 10:40 GMT+02:00

    We have identified an issue with the button automation. When users click on the button inside message templates that they have received then they either get a catch all message or no message at all from the service.

    This is impacting all clients. We are working hard to get it resolved.

Infrastructure rolling restart

Resolved | Jul 04, 2021 | 00:02 GMT+02:00

The upgrades have completed and everything's been operating normally since 19:20 UTC.

+Show history

  • Open | Jul 03, 2021 | 20:49 GMT+02:00

    Due to an unannounced service upgrade, the cluster powering app.turn.io was restarted. This may cause some intermittent interruptions in messaging and delays in messages being available in search.

    We'll update this status update once completed.

Errors in threads feature in turn.io

Resolved | Jun 25, 2021 | 14:08 GMT+02:00

The issue related to threads is now resolved. All features are working as expected.

+Show history

  • Open | Jun 25, 2021 | 10:58 GMT+02:00

    We have found an issue that is affecting threads feature. This is impacting some of the clients on app.turn.io. Issues such as saving threads and updating threads is associated with it. We are working tirelessly to get it resolved as quickly as possible.

Threads page not available

Resolved | Jun 21, 2021 | 12:00 GMT+02:00

We have resolved the issue related to threads page. Threads page is working as expected for all clients.

+Show history

  • Open | Jun 21, 2021 | 09:54 GMT+02:00

    We have identified an issue with the Threads automation page. Threads page is giving a white screen and is failing to load. This is impacting all clients.

    Our teams are working on the issue. We hope to resolve it soon.

Error while sending media messages and templated messages with buttons.

Resolved | May 13, 2021 | 19:13 GMT+02:00

The issue of sending media messages and templated messages with button is now resolved. All systems are working as expected.

+Show history

  • Monitoring | May 13, 2021 | 12:10 GMT+02:00

    The issue persists at WhatsApp's end. Messages templates with Buttons and Media messages are impacted due to this outage. We are closely following the updates from WhatsApp.

Turn/Collection/Automation not loading

Resolved | Apr 01, 2021 | 13:32 GMT+02:00

The issue pertaining to loading of conversations is now resolved. All features are working as expected.

+Show history

  • Open | Apr 01, 2021 | 12:51 GMT+02:00

    We have encountered an error at our end due to which either turn is not loading or the collections/automations are not available. This is impacting all clients on app.turn.io. It is not impacting private clouds.

    Our engineers are working to get it resolved as soon as possible.

Application loading issues due to outages in content distribution network

Resolved | Mar 22, 2021 | 02:21 GMT+01:00

Content distribution network outage ended. All systems are working as expected.

+Show history

  • Open | Mar 19, 2021 | 09:04 GMT+01:00

    The content distribution network that is used to serve the Turn frontend application is currently experiencing a major outage. This outage might impact some of our clients.

    Some of our clients might not be able to load the turn application. We are monitoring the situation.

Message delivery fails for clients on Infobip

Resolved | Mar 16, 2021 | 12:09 GMT+01:00

Clients on Infobip can now send and receive messages. The issue is resolved now.

+Show history

  • Open | Mar 16, 2021 | 08:33 GMT+01:00

    We have observed that the message delivery for clients on Infobip BSP fails. This is happening for all inbound and outbound messages. The issue is impacting only those clients who are on Infobip.

    We are working to resolve the issue as soon as possible.

Maintenance Update

Resolved | Mar 04, 2021 | 06:39 GMT+01:00

We have upgraded our search infrastructure and message history is now available for 12 months.

+Show history

  • Resolution in progress | Feb 22, 2021 | 07:08 GMT+01:00

    We have restored the chat history for past 12 months for all app.turn.io clients. The backfill for private cloud clients are now in progress and should be completed in the next 2 days. We have also upgraded our search infrastructure. This means that the time taken for execution of search query is reduced. This also means that the performance of loading conversations, receiving and sending messages is also improved.

  • Resolution in progress | Feb 17, 2021 | 12:38 GMT+01:00

    We have started the process to restore your full message history. As a result, search results and totals shown in Collections may vary more than expected. We will update you once the history is fully restored.

  • Resolution in progress | Feb 11, 2021 | 09:11 GMT+01:00

    We're performing a system upgrade in response to performance issues with our search infrastructure. This will temporarily limit Search history to active users to the last 6 weeks for all our clients. As a result, the totals shown in Collections will only reflect users who have interacted with your service since 18 December 2020. No other features will be impacted except for Collections and Search.

    To can learn more visit:
    https://learn.turn.io/product-updates/10-february-2021-temporary-search-limitations-while-we-do-a-search-feature-upgrade

Automation count limited to 50 rules

Resolved | Feb 24, 2021 | 16:08 GMT+01:00

All automations are now available.

+Show history

  • Open | Feb 23, 2021 | 16:30 GMT+01:00

    We have found an issue in our system which is causing availability of only 50 automation rules. All the rules that are not available are working fine for the users but those rules are not visible.

    This is impacting all clients including clients on private clouds.

    We are working to get it resolved soon.

Chats/Conversations not loading

Resolved | Feb 04, 2021 | 11:06 GMT+01:00

We have deployed a fix for this issue. The changes that we have done in our infrastructure have not only solved the loading conversations/chat issue but also reduce the time of loading chats.

Chats/conversations should now be working properly for all clients including clients on private clouds.

Error in Loading of Conversation

Resolved | Jan 29, 2021 | 11:00 GMT+01:00

Conversation are loading for all clients now.

+Show history

  • Monitoring | Jan 26, 2021 | 11:47 GMT+01:00

    We have applied a fix for the issues of loading conversation and display of chats on dashboard. All clients can now have a smooth loading of their conversations and should now have all the chats displayed in their dashboards.

    We are, however, keeping this issue under monitoring.

  • Open | Jan 25, 2021 | 14:27 GMT+01:00

    The issues of loading conversation and availability of chats in dashboard are now impacting all clients including clients on private clouds.

    Our engineering teams are constantly working on a fix.

  • Resolution in progress | Jan 25, 2021 | 12:49 GMT+01:00

    We are making progress on the issue of loading conversations. Currently, conversations are loading fine for all clients.

    There is however a related issue of availability of new chats in the dashboard. New Chats are not reflected in the client's dashboard. This is affecting all clients on app.turn.io.

    We are as eager as you are to solve the issue and we are working tirelessly to achieve that.

  • Open | Jan 25, 2021 | 09:46 GMT+01:00

    Issue of loading collections still persists for all the clients on app.turn.io. We are working on it to get it resolved.

  • Resolution in progress | Jan 22, 2021 | 08:54 GMT+01:00

    We have applied some changes to improve the performance. We are keeping it under watch and working on a fix.

  • Open | Jan 20, 2021 | 07:01 GMT+01:00

    Conversations fails to load currently. This issue is affecting most of the clients on app.turn.io . This issue is not affecting the clients with private clouds. We are working hard to resolve it.

  • Resolution in progress | Jan 19, 2021 | 06:28 GMT+01:00

    Collections are now loading but we are experiencing a degraded performance. It is taking more time than usual to load collection. We are working hard to get it resolved.

  • Monitoring | Jan 18, 2021 | 12:48 GMT+01:00

    We have corrected our systems and now the conversations are loading properly. We are monitoring our system for next few hours.

  • Open | Jan 18, 2021 | 08:01 GMT+01:00

    We have found an issue with app.turn.io which is primarily affecting loading of conversations in most of the whatsapp Bot lines.

    We are working hard to get it resolved as quickly as possible.

Automatic logout

Resolved | Jan 05, 2021 | 17:24 GMT+01:00

We've been observing this issue over the holidays and it is now resolved.

+Show history

  • Open | Dec 23, 2020 | 12:32 GMT+01:00

    If you continue to have problems logging and get an error about an "invalid token" please delete the browser cookies for https://app.turn.ioa and https://whatsapp.turn.io and refresh

  • Open | Dec 23, 2020 | 11:06 GMT+01:00

    We've applied some extra security policy's to Turn. If you are finding yourself unable to connect, or disconnected, you need to logout and log back in again.

Messaging interruptions

Resolved | Dec 23, 2020 | 12:50 GMT+01:00

This issue is now resolved.

We've identified the root cause to relate to big query synchronisation for a number of clients experiencing above average messaging throughputs. We've disabled the big query synchronisation for one account and are working on a permanent fix.

+Show history

  • Open | Dec 23, 2020 | 12:31 GMT+01:00

    We're experiencing some interruptions with message delivery. We are investigating the root cause.

Backlog on search

Resolved | Dec 22, 2020 | 11:22 GMT+01:00

The issue was resolved, all backlogged messages that were still waiting to be made available in the search and collections results have been processed.

+Show history

  • Open | Dec 21, 2020 | 14:12 GMT+01:00

    We're seeing a backlog build up on our search infrastructure. This can result in a delay in messages and chats becoming visible in collections. We're working on resolving this.

    Messaging is unaffected by this.

Message processing interruptions

Resolved | Dec 18, 2020 | 15:56 GMT+01:00

The issue has been resolved.

+Show history

  • Investigating | Dec 18, 2020 | 11:09 GMT+01:00

    While messages are being processed as normal, they may not be showing up in search results or in the chat list yet.

    We are continuing to investigate the issues with the search infrastructure and will update here as soon as we know more.

  • Open | Dec 18, 2020 | 10:26 GMT+01:00

    We're experiencing down time due to problems with indexing of messages on our search infrastructure.

    We're investigation and will update as soon as we know more.

Google login issues

Resolved | Dec 14, 2020 | 15:21 GMT+01:00

Google seems to have resolved most of their issues.

+Show history

  • Open | Dec 14, 2020 | 13:13 GMT+01:00

    Google is having a global outage and Turn relies on Google for login and account management.

    While Google is having these problems, you may face issues while logging into any Turn instance.

Automation issue

Resolved | Nov 27, 2020 | 15:23 GMT+01:00

The issue has been resolved.

+Show history

  • Open | Nov 27, 2020 | 11:42 GMT+01:00

    We've found an issue with automations that is impacting numbers who have more than 50 pieces of content loaded in Turn. We're busy addressing this issue.

Database changes interrupting message processing.

Resolved | Nov 10, 2020 | 16:42 GMT+01:00

The issue has been resolved for all Turn installations.

+Show history

  • Monitoring | Nov 10, 2020 | 13:11 GMT+01:00

    The issue is resolved on app.turn.io, we're monitoring the situation on the private cloud installations.

Downtime

Resolved | Oct 21, 2020 | 14:13 GMT+02:00

The issue has been resolved for the 2 clusters impacted. The underlying technical root cause was a database migration causing a table lock on one of the applications' main tables. This was further complicated by automated database cleanup processes happening on the same table a the same time.

Our apologies for the downtime experienced.

+Show history

  • Investigating | Oct 21, 2020 | 12:44 GMT+02:00

    The fix we've applied has not yet had it's intended effect. We're continuing to investigate.

  • Resolution in progress | Oct 21, 2020 | 11:49 GMT+02:00

    A fix is being released to the production environments impacted.

  • Open | Oct 21, 2020 | 11:17 GMT+02:00

    Unfortunately we pushed a faulty release to 2 of our production clusters. This is causing downtime for https://app.turn.io. We are busy issuing a fix and the systems should be online again shortly.

    Our sincere apologies for this inconvenience, we will provide updates here as we address the underlying issue.

Problems processing status updates

Resolved | Oct 08, 2020 | 14:07 GMT+02:00

The issue has been resolved.

+Show history

  • Open | Oct 08, 2020 | 13:03 GMT+02:00

    We've found an issue where Turn is not accepting status updates on messages. This can mean that you'll experience a delay in seeing your messages marked as read, sent, or delivered. We're busy addressing this issue.

Maintenance update

Resolved | Sep 15, 2020 | 10:47 GMT+02:00

All system upgrades have been applied.

+Show history

  • Open | Sep 15, 2020 | 10:14 GMT+02:00

    We're performing a system upgrade in response to Google's https://cloud.google.com/kubernetes-engine/docs/security-bulletins#gcp-2020-012 security bulletin on all our Google Cloud Platform hosted systems. This will involve various restarts across the platforms. Any messages received during a restart will be handled immediately as the restarts of the various components are completed.

Messages not being delivered to Turn

Resolved | Sep 15, 2020 | 10:07 GMT+02:00

We've observed this for a few days and it is now resolved.

+Show history

  • Monitoring | Sep 11, 2020 | 13:38 GMT+02:00

    We've put some more fixes in places and are observing the system's behaviour. This issue seems to be related to automatic upgrades performed by the hosting services that app.turn.io is using. This is not impacting any of the private cloud setups.

  • Investigating | Sep 11, 2020 | 12:01 GMT+02:00

    The fix we put in place is no longer resolving this issue we are experiencing with database connections. We're investigating.

  • Resolved | Sep 11, 2020 | 10:43 GMT+02:00

    The issue is resolved. We're continuing to monitor the systems.

  • Open | Sep 11, 2020 | 10:31 GMT+02:00

    We've found an issue with a number of numbers unable to receive messages from WhatsApp. We've identified the issue and are busy addressing it.

    Any messages currently pending delivery will be processed shortly.

Collections not loading intermittently

Resolved | Aug 25, 2020 | 09:23 GMT+02:00

Collection search indexing has been optimised to reduce load on the search backend. Search query and collection performance has been greatly improved as a result.

We're continuing to work on improvements to prevent this happening in future - in particular we're changing the way All Conversations is architected so that it is much faster and doesn't rely on search

+Show history

  • Monitoring | Aug 25, 2020 | 09:22 GMT+02:00

    Collection search indexing has been optimised to reduce load on the search backend. Search query and collection performance has been greatly improved as a result.

    We're continuing to work on ways to prevent this occurring in future including changing the way All Conversations is architected so that it is much faster and doesn't rely on search.

  • Resolution in progress | Aug 19, 2020 | 14:35 GMT+02:00

    To alleviate some of the load we'll be temporarily be turning off the total user count display for collections.

  • Resolution in progress | Aug 19, 2020 | 13:39 GMT+02:00

    Currently, Collections rely on Search functionality to find conversations that match the Collection criteria. The All Conversations Collection, in particular, puts a high load on Search, slowing down the performance across all other Collections.

    We're busy working on a change to how the All Conversation Collection functions to ensure that is performs faster.

  • Investigating | Aug 18, 2020 | 12:42 GMT+02:00

    Some users are experiencing issues where Collections are loading a blank screen, or partial screen.

    We're exploring different reasons this may be happening.

Status: Content Issue | 19 August 2020 | 10:58 (GMT+2)

Resolved | Aug 19, 2020 | 13:38 GMT+02:00

Content is now behaving as usual on app.turn.io

+Show history

  • Resolution in progress | Aug 19, 2020 | 10:58 GMT+02:00

    We found an issue where content is no longer editable due to a change we made in the underlying content format.

    This has now been reverted. Please do a hard refresh to ensure you see the changes.

Inbound media not displaying

Resolved | Jul 31, 2020 | 17:25 GMT+02:00

A new release is being deployed to all the clusters, all inbound media for Infobip numbers will be working again as expected within a few minutes.

+Show history

  • Resolution in progress | Jul 31, 2020 | 12:23 GMT+02:00

    QA passed, we'll be releasing to all our production environments shortly.

  • Resolution in progress | Jul 31, 2020 | 10:55 GMT+02:00

    We've identified the issue and have a fix in place. We're currently testing it in our QA environment.

  • Identified | Jul 31, 2020 | 10:28 GMT+02:00

    This seems to be limited to numbers backed by Infobip. We're continuing to dig into this issue.

  • Open | Jul 31, 2020 | 09:03 GMT+02:00

    We're seeing an issue where media received via WhatsApp is not displaying properly in the UI nor accessible via the API.

    We are busy investigating and will provide updates as soon as possible.

Automation queue backlogged

Resolved | Jul 28, 2020 | 17:27 GMT+02:00

Issue is resolved

+Show history

  • Closed | Jul 28, 2020 | 17:26 GMT+02:00

    The issue has been resolved.

  • Resolution in progress | Jul 28, 2020 | 15:24 GMT+02:00

    We've indentified one possible root cause which could result in processed messages being rescheduled for processing again by the automation system.

  • Resolution in progress | Jul 28, 2020 | 14:47 GMT+02:00

    The backlog of background jobs has been processed and things should be snappy again. We're continuing to investigate the root cause of this issue.

  • Identified | Jul 28, 2020 | 14:37 GMT+02:00

    We've identified a situation where a number is constantly rescheduling new jobs in the background job queue causing it to grow constantly.

  • Open | Jul 28, 2020 | 14:08 GMT+02:00

    Due to a problematic release, there are quite a few messages pending to be processed and still due for delivery. We are working through the backlog and services relying on the automation system should be responsive again shortly.

Infrastructure capacity

Resolved | Jul 22, 2020 | 14:20 GMT+02:00

This issue is now resolved. Please get in touch with us if you are having issues with conversations or messages not being visible in the UI search results or collections.

+Show history

  • Monitoring | Jul 22, 2020 | 13:41 GMT+02:00

    Our search infrastructure upgrade has completed and we're processing a backlog of conversations and messages needing to be processed by the search backend. Once they've been processed they will show up in your search results and collections overviews. We're expecting this to be completed within the next hour.

  • Open | Jul 22, 2020 | 11:03 GMT+02:00

    We've reprovisioned the search infrastructure and are re-indexing the chat histories for all clients. While this in progress clients may have an incomplete view of the chats in search results and collections.

  • Open | Jul 20, 2020 | 14:22 GMT+02:00

    We're noticing we're hitting some infrastructure capacity limits on our app.turn.io hosting environment. We're busy adding more capacity to meet the demand.

Elasticsearch degraded performance

Resolved | Jun 30, 2020 | 12:23 GMT+02:00

This issue has for now been resolved.

+Show history

  • Monitoring | Jun 29, 2020 | 12:29 GMT+02:00

    Services are operating as normal again. The search infrastructure is doing better but the underlying cause is not fully resolved yet. We're working on a more permanent solution.

  • Open | Jun 29, 2020 | 11:22 GMT+02:00

    Our Elasticsearch installation is suffering from degraded performance. This is causing the UI to load slowly for many users. We are working on restoring service as soon as possible.

    Messaging services should be operational as normal.

Message delivery interruptions

Resolved | Jun 10, 2020 | 12:06 GMT+02:00

This issue has been resolved.

+Show history

  • Open | Jun 10, 2020 | 11:52 GMT+02:00

    Due to a new software release, some messages are not being delivered to some countries.
    A fix is being released and should be resolved shortly.

Status: Investigating | 30/05/2020 | 13:12 CET

Resolved | May 30, 2020 | 13:26 GMT+02:00

We've updated how we route traffic to whatsapp.turn.io and the issue has been resolved by issuing a different SSL certificate.

+Show history

  • Open | May 30, 2020 | 13:12 GMT+02:00

    We have found an issue with an SSL certificate's authority certificate having expired which is causing some downtime for some clients.

    We're working on resolving this.

Turn Status: Dashboard loading slowly for some clients | 7 February 2020

Resolved | Feb 27, 2020 | 15:43 GMT+01:00

This issue has been resolved.

+Show history

  • Resolution in progress | Feb 07, 2020 | 12:44 GMT+01:00

    For clients with many collections spanning hundreds of thousands of monthly active users the Turn dashboard page was having trouble loading. We isolated the issue to collections taking longer to load than could reasonably be expected.

    We are working on a fix to address this issue for these clients.

Irregular media attachement behaviour

Resolved | Feb 26, 2020 | 12:21 GMT+01:00

This issue has been resolved.

+Show history

  • Monitoring | Feb 24, 2020 | 13:29 GMT+01:00

    We're investigating some issues reported with images not being delivered via Infobip.

  • Monitoring | Feb 12, 2020 | 08:48 GMT+01:00

    We are continuing to monitor media attachment behaviour over the next few days.

  • Investigating | Jan 31, 2020 | 10:48 GMT+01:00

    We have found an issue where attachments such as location pins and audio files are displaying incorrectly in the Turn UI.

    The Turn team is working to get it resolved as quickly as possible.

Turn Status: Background job worker running out of disk space | 7 February 2020

Resolved | Feb 12, 2020 | 08:47 GMT+01:00

Faktory has been behaving as expected over the last 5 days and therefore this incident is considered resolved.

+Show history

  • Monitoring | Feb 07, 2020 | 16:19 GMT+01:00

    A temporary fix was put in place at 9:07 UTC. A permanent fix will be deployed shortly.

  • Monitoring | Feb 07, 2020 | 12:43 GMT+01:00

    We have found an issue with Turn and are working tirelessly to get it resolved as quickly as possible.Turn off-loads the delivery of webhooks and indexing of messages and conversations to a background process. This allows Turn’s APIs and frontend to remain performant by delegating tasks that do not require immediate processing, or tasks which are allowed to take a little longer to complete, and to continue working in the background while Turn continues to focus on message processing.

    This morning the piece of Turn infrastructure that is responsible for accepting and scheduling these tasks ran into its maximum allowance of disk space. As a result, it intermittently refused to accept more tasks as it briefly had no disk space to store them.

    The impact on clients was that webhook deliveries were delayed and the search and conversation listings were at times showing outdated information.

    We increased the disk space allocation for the background worker and we have reindexed all the messages and conversations for all clients to make sure the frontend is displaying the latest messages and updated conversations again.

    We are putting measures in place to make sure the disk space allocations increase as our clients’ monthly active users and message volumes increase.

Status Page powered by Admin Labs