Executive Summary Event Description Accounts located in the US East and EU Central regions experienced ingestion delays with some of their data, leading to fresh data not being visible on Logz.io. Factors Leading to Event As a result of overallocation of AWS EC2 capacity, required resources were hitting EC2 limits and failed to launch. This also caused a networking issue which caused limited data ingestion capacity. Impact There was a 3.5 hour period where some fresh data was not ingested. For all customers in the US and for a vast majority of the customers in the EU this latency was recovered 3 hours later. For a small subset of customers in EU, this was recovered 24 hours later. Customers who used non-resilient data shipping mechanisms may have experienced data loss during that time. Some data was recovered and appended to the index of the next day Customers who are missing any should reach out to Logz.io support team Time (Aug 28, IDT) Event 12:50 Production on-call engineer started to get alerts regarding failure of the data ingestion pipeline 13:03 Issue was escalated to a Senior engineer who started investigating the issue. A large group of Operations and Software Development engineering joined the research 14:00 Networking issue detected and the work on resolution has started 14:40 All defective nodes were removed, and networking issues were resolved 15:12 First “fresh” resources are launched. Fresh datcoming in 16:12 Issue resolved. Started adding the required resources to handle the accumulated lag. 19:15 Lag has been consumed for 99% of the accounts 21:00 Lag consumed for all the accounts Corrective Actions Immediate Term Work with AWS to make sure that there is a 24x7 open communication channel to handle such incidents along with the ability to manually “override” any capacity constraints Review EC2 limits across our cloud operations Medium Term Improve our automation capabilities to launch needed resources faster in a 100% automated manner Further deepen our monitoring to detect these capacity issues earlier with more precision
Logz.io is committed to providing a stable service to its customers and we are fully committed to getting your data in time with minimal delays.
Logz.io views this incident very seriously due to its impact in terms of breadth and time.
Thanks to our robust disaster recovery procedures, even in this magnitude of an event we were able to start ingesting fresh data in a matter of a couple of hours
We have since been working with AWS around the clock to ensure that such a failure will not happen again.