Log ingestion delays

Incident Report for logz.io

Postmortem

At 2:25am UTC, there was a problem with one of our ingestion pipelines and queuing system for a specific TCP input. We're recovering all available backed up log data from backup (between 2:25-2:50) while some data might be missing for that time frame. We're still finalizing our root cause analysis but we suspect the cause of the issue is a failure of EC2 instances in two different AZs which caused the queuing system to become non functional for some pipelines including some backup procedure. We'll be applying some changes over the weekend to make sure this cannot happen again

Posted Jan 29, 2016 - 10:26 IST

Resolved

See postmortem.
Posted Jan 29, 2016 - 09:54 IST

Identified

Log ingestion is back to normal. We're now filling in gaps in log ingestion.
Posted Jan 29, 2016 - 05:00 IST

Investigating

We're having log ingestion delays on specific ingestion pipeline
Posted Jan 29, 2016 - 04:24 IST