Turnio - Automation queue backlogged – Incident details

All systems operational

Automation queue backlogged

Resolved
Major outage
Started about 5 years agoLasted about 3 hours
Updates
  • Update
    Update
    Issue is resolved
  • Resolved
    Resolved
    The issue has been resolved.
  • Update
    Update
    We've indentified one possible root cause which could result in processed messages being rescheduled for processing again by the automation system.
  • Update
    Update
    The backlog of background jobs has been processed and things should be snappy again. We're continuing to investigate the root cause of this issue.
  • Identified
    Identified
    We've identified a situation where a number is constantly rescheduling new jobs in the background job queue causing it to grow constantly.
  • Update
    Update
    Due to a problematic release, there are quite a few messages pending to be processed and still due for delivery. We are working through the backlog and services relying on the automation system should be responsive again shortly.
  • Investigating
    Investigating