On Wednesday, March 16, several of our customers experienced delays in log processing. Newer logs were not processed on time, causing a service interruption. We apologize for this service interruption and wanted to share our root cause analysis with you.
At 11:00 GMT, an alert was triggered, indicating a slowdown in data processing. Upon further investigation, the on-call team realized that the Logz.io distributed data parsing and ingestion system had significantly slowed down and was producing a slow output. Further investigation showed that specific logs had been exposed to a bug in the Logz.io log parsers, causing the severe slowdown.
The Logz.io log parsers work in a distributed manner at a very high scale, making the task of finding the exact logs that contain information on the reason for the slowdown a complicated task.
At 11:25 GMT, the team started moving customers to different queues within our queuing system, allowing most of our customers to receive updated logs. Our R&D teams rewrote one of the log parsers to allow for the detection of those logs that contained the relevant information. This tasks required several iteration until it worked and was able to move to production.
At 18:20 GMT, this code was deployed and the problematic log type was detected and blocked.
By that time, the queuing system held a queue of billions of logs, it took another 1-2 hours until all customers were receiving told log data.
We continued to monitor all systems closely throughout the following hours, gathering as much electronic and human information about the incident as possible for further analysis and improvement.
Logz.io is having multiple layers of availability and backups, all data was retained and ingested back into the system.
Conclusions and Action Items 1. The log parsing bug was detected and fixed.
There was a need to develop tools to quickly isolate log lines that were causing the parsing issues and block them from interfering with the ongoing ingestion process. These tools were developed, and if such parsing bugs will ever occur again in the future, they will be isolated easily.
We have enhanced our internal automation procedure to isolate any instances of slow log processing so that we can limit the instances only to those specific customers. This will ensure that no one else would be affected.