Root Cause Analysis
March 29, 2022
Event Description Service Outage in multiple regions. Application was unavailable. Latency in log ingestion. Factors Leading to Event A human error led to an erroneous change in our service network configuration. Logz.io aims to tightly protect all its essential configuration data, and to have the proper disaster-recovery procedures in place in the unlikely event that the protection was breached. In this case due to a human error, a network configuration modification which should have been applied to an isolated non-Production environment, errantly propagated into Production, and caused a major outage. In such extreme circumstances all internal network connectivity is lost. After restoring the configuration, all the services had to be restarted, to enable them to re-establish network connectivity. The root cause was identified very early in the incident. Immediate term actions are already in the process of being taken to prevent modification of sensitive configurations manually, and to improve the speed of the configuration recovery. Other action times are in the works to enable faster recovery of our more crucial central services. Impact The issue that started on March 29th was characterized by: App unavailability for 2 hours in us-east region, 5 hours in eu-central region, and 7.5 hours in the other regions. Indexing lag in a few of the regions which was fully consumed after 6 hours Missing real-time data in several Metrics accounts in us-east region
Time (UTC) Event Mar 29, 22:09 Multiple alerts across multiple regions Mar 29, 22:14 On–call team is reviewing the issue; Due to its severity issue immediately escalated, and an all-hands-on-deck alert is triggered. Mar 29, 22:20 Root cause of the issue identified Mar 29, 23:45 Basic network configuration restored. Disaster recovery procedure of service recovery is initiated. Mar 30, 00:19 Group of critical services, including the authentication one, is up and running in us-east. Application is performing in us-east. Ingestion operating at sub-optimal capacity Mar 30, 00:30 Service recovery procedure rolling out to all regions Mar 30, 04:00 Ingestion in all regions back to full capacity Mar 30, 08:40 Incident is fully recovered and all delays have been fully ingested
Immediate Term A strict manual change process was deployed, to disable any changes in critical data in any environment without the approval of a Director level manager. Continuous backup of network configurations and developing the automation for its recovery. Losing all connectivity disabled us from using our internal escalation system, which if could be accelerated even by a few minutes would have made a difference. We are adding a layer to ensure that this escalation can be performed outside of the domain of our service. Short and medium Term The recovery time in this case needs to be accelerated. Most of our recovery process are fully automated however, this specific issue required a combination of manual (hence time-consuming, and error-prone) and automated actions. While more of these actions are continuously being automated, we will continue working on this until we can execute all the required actions without human intervention. This automation effort is already in motion with the recovery of our most critical services At the same time we are allocating more time and resources to train the team, including simulated “war days” and more documentation.
Logz.io is committed to providing stable service to its customers and we are fully committed to getting your data in time with minimal delays. We keep working on improving our disaster recovery processes to further reduce downtime in extreme situations.We’re going to invest significant resources to make sure such outages will not happen again.