Turnio - Automation queue backlogged – Incident details

All systems operational

Automation queue backlogged

ResolvedMajor outage3 hours 19 minutes
  • UpdateUTC
    Issue is resolved
  • ResolvedUTC
    The issue has been resolved.
  • UpdateUTC
    We've indentified one possible root cause which could result in processed messages being rescheduled for processing again by the automation system.
  • UpdateUTC
    The backlog of background jobs has been processed and things should be snappy again. We're continuing to investigate the root cause of this issue.
  • IdentifiedUTC
    We've identified a situation where a number is constantly rescheduling new jobs in the background job queue causing it to grow constantly.
  • UpdateUTC
    Due to a problematic release, there are quite a few messages pending to be processed and still due for delivery. We are working through the backlog and services relying on the automation system should be responsive again shortly.
  • InvestigatingUTC