Almost all customers now have recent logs, last few are catching up quickly.
Posted Jun 15, 2019 - 12:19 IDT
Update
We are continuing to monitor for any further issues.
Posted Jun 15, 2019 - 10:08 IDT
Monitoring
The fixes we have applied are working as expected, and ingestion is catching up. We are continuing to monitor the recovery progress and will continue to update.
Posted Jun 15, 2019 - 08:48 IDT
Update
We are continuing our work on resuming full operations to the customers of the Frankfurt region that are affected. We will update as we progress.
Posted Jun 15, 2019 - 08:24 IDT
Update
We are continuing our work on resuming full operations to the customers of the Frankfurt region that are affected. We will update as we progress.
Posted Jun 15, 2019 - 05:22 IDT
Identified
We have identified another issue for some of our customer in the Frankfurt region, and are working to find a resolution
Posted Jun 15, 2019 - 04:00 IDT
Monitoring
Most of the affected customers have all updated logs and we are continuing to monitor recovery for all customers from the group that was affected in the Frankfurt region
Posted Jun 15, 2019 - 02:54 IDT
Update
As the problem is being resolved, currently the issue is ingestion delays, which also affect Alerting. We are continuing to work on resolving all the remaining issues for this subset of customers in the Frankfurt region.
Posted Jun 15, 2019 - 01:43 IDT
Identified
We have identified the root cause of the degradation, and are working towards implementing the fix. Some subsystems like API and User Interfaces are expected to recover first.
Posted Jun 15, 2019 - 01:08 IDT
Investigating
A subset of our Frankfurt region might experience elevated error rates when searching in Kibana or the API. This also might affect altering accuracy and log ingestion delays. Our engineers have identified the issue and working fiercely to resolve it.
Posted Jun 15, 2019 - 01:02 IDT
This incident affected: Indexing, User Interface, Alerting, and API.