Indexing and API degradation for some of the customers in the Frankfurt region

Incident Report for logz.io

Resolved

Almost all customers now have recent logs, last few are catching up quickly.
Posted Jun 15, 2019 - 12:19 IDT

Update

We are continuing to monitor for any further issues.
Posted Jun 15, 2019 - 10:08 IDT

Monitoring

The fixes we have applied are working as expected, and ingestion is catching up. We are continuing to monitor the recovery progress and will continue to update.
Posted Jun 15, 2019 - 08:48 IDT

Update

We are continuing our work on resuming full operations to the customers of the Frankfurt region that are affected. We will update as we progress.
Posted Jun 15, 2019 - 08:24 IDT

Update

We are continuing our work on resuming full operations to the customers of the Frankfurt region that are affected. We will update as we progress.
Posted Jun 15, 2019 - 05:22 IDT

Identified

We have identified another issue for some of our customer in the Frankfurt region, and are working to find a resolution
Posted Jun 15, 2019 - 04:00 IDT

Monitoring

Most of the affected customers have all updated logs and we are continuing to monitor recovery for all customers from the group that was affected in the Frankfurt region
Posted Jun 15, 2019 - 02:54 IDT

Update

As the problem is being resolved, currently the issue is ingestion delays, which also affect Alerting. We are continuing to work on resolving all the remaining issues for this subset of customers in the Frankfurt region.
Posted Jun 15, 2019 - 01:43 IDT

Identified

We have identified the root cause of the degradation, and are working towards implementing the fix. Some subsystems like API and User Interfaces are expected to recover first.
Posted Jun 15, 2019 - 01:08 IDT

Investigating

A subset of our Frankfurt region might experience elevated error rates when searching in Kibana or the API. This also might affect altering accuracy and log ingestion delays.
Our engineers have identified the issue and working fiercely to resolve it.
Posted Jun 15, 2019 - 01:02 IDT