Turnio - Notice history

All system upgrades have been applied.

We're performing a system upgrade in response to Google's https://cloud.google.com/kubernetes-engine/docs/security-bulletins#gcp-2020-012 security bulletin on all our Google Cloud Platform hosted systems. This will involve various restarts across the platforms. Any messages received during a restart will be handled immediately as the restarts of the various components are completed.

We've observed this for a few days and it is now resolved.

We've put some more fixes in places and are observing the system's behaviour. This issue seems to be related to automatic upgrades performed by the hosting services that app.turn.io is using. This is not impacting any of the private cloud setups.

The fix we put in place is no longer resolving this issue we are experiencing with database connections. We're investigating.

The issue is resolved. We're continuing to monitor the systems.

We've found an issue with a number of numbers unable to receive messages from WhatsApp. We've identified the issue and are busy addressing it. Any messages currently pending delivery will be processed shortly.

Content is now behaving as usual on app.turn.io

We found an issue where content is no longer editable due to a change we made in the underlying content format. This has now been reverted. Please do a hard refresh to ensure you see the changes.

Collection search indexing has been optimised to reduce load on the search backend. Search query and collection performance has been greatly improved as a result. We're continuing to work on improvements to prevent this happening in future - in particular we're changing the way All Conversations is architected so that it is much faster and doesn't rely on search

Collection search indexing has been optimised to reduce load on the search backend. Search query and collection performance has been greatly improved as a result. We're continuing to work on ways to prevent this occurring in future including changing the way All Conversations is architected so that it is much faster and doesn't rely on search.

To alleviate some of the load we'll be temporarily be turning off the total user count display for collections.

Currently, Collections rely on Search functionality to find conversations that match the Collection criteria. The All Conversations Collection, in particular, puts a high load on Search, slowing down the performance across all other Collections. We're busy working on a change to how the All Conversation Collection functions to ensure that is performs faster.

Some users are experiencing issues where Collections are loading a blank screen, or partial screen. We're exploring different reasons this may be happening.

A new release is being deployed to all the clusters, all inbound media for Infobip numbers will be working again as expected within a few minutes.

QA passed, we'll be releasing to all our production environments shortly.

We've identified the issue and have a fix in place. We're currently testing it in our QA environment.

This seems to be limited to numbers backed by Infobip. We're continuing to dig into this issue.

We're seeing an issue where media received via WhatsApp is not displaying properly in the UI nor accessible via the API. We are busy investigating and will provide updates as soon as possible.

Issue is resolved

The issue has been resolved.

We've indentified one possible root cause which could result in processed messages being rescheduled for processing again by the automation system.

The backlog of background jobs has been processed and things should be snappy again. We're continuing to investigate the root cause of this issue.

We've identified a situation where a number is constantly rescheduling new jobs in the background job queue causing it to grow constantly.

Due to a problematic release, there are quite a few messages pending to be processed and still due for delivery. We are working through the backlog and services relying on the automation system should be responsive again shortly.

This issue is now resolved. Please get in touch with us if you are having issues with conversations or messages not being visible in the UI search results or collections.

Our search infrastructure upgrade has completed and we're processing a backlog of conversations and messages needing to be processed by the search backend. Once they've been processed they will show up in your search results and collections overviews. We're expecting this to be completed within the next hour.

We've reprovisioned the search infrastructure and are re-indexing the chat histories for all clients. While this in progress clients may have an incomplete view of the chats in search results and collections.

We're noticing we're hitting some infrastructure capacity limits on our app.turn.io hosting environment. We're busy adding more capacity to meet the demand.