Logz.io is fully committed to maintaining the highest levels of SLAs.

We understand that many businesses rely on logz.io for mission-critical information, and as such the continuous uptime of our system is crucial.

Events on Nov 28 - all time UTC:

1645: Our monitoring systems report a degradation in the performance of logins into our system.

1645: On-Call starts investigating

16:50: On-call team identified a 3rd party authentication service we use, named Auth0 as the source of login delays. Users logged in over the last 24 were able to log in but others reported long login times and timeout message when trying to login.

1700: Our monitoring systems report a further deterioration in the authentication system and failure in login attempts.

1713: We receive communication from our authentication service provider - Auth0 - that it is has identified a problem in their service

1715: We started communication with Auth0 to understand the magnitude of the problem and what can be done to alleviate it

1830: Situation alleviated. Dramatic reduction in failed login attempts

1930: Issue fully resolved

Root Cause Analysis:

The logz.io system uses a third-party app service - Auth0 - as an authentication platform. Logz.io relies on Auth0 SLA of 99.9 which was not met during this incident. The Auth0 incident is documented here.

The incident had no impact on customer data.

Action Items:

• Immediate term: work with our authentication provider to increase SLA and to ensure that the redundancy mechanisms provided by it are rigorous. Those mechanisms failed to kick in during this incident despite the fact that we are subscribers to its top-tier service level.
• Midterm: Deliver an authentication service which meets our SLA’s and supports the high standards we commit to our customers. We will revisit the decision to use Auth0 and either select a more performant service or develop a fallback mechanism