Summary
Vivi undertook an upgrade of its AWS infrastructure on the evening of Monday 7th May at 1800 AEST both to increase capacity and keep up with the demands for Vivi’s services and to upgrade the version of the software frameworks that the Vivi admin portal is built on to ensure that they remained under support as the current version in use was due to be deprecated. While internal testing showed the upgrade to be successful we encountered an incident at 0840 AEST when customer load started to hit the systems which was partially resolved at 1007 and completely resolved by 1209, following the resolution there was a further short intermittent outage at 1404 AEST when a database server had to be restarted. The incident caused intermittent access to the admin portal and customers with particular LDAP configurations not being able to authenticate.
In summary there were three underlying root causes:
A database became overloaded causing intermittent issues with the admin portal, this would have affected all customers causing issues with login attempts to both the admin portal and the client and therefore affecting screen sharing for users that weren’t already logged in.
An incompatibility between our LDAP authentication library and the new version of the software framework we had upgraded to that only occurred within a specific code path. This would have affected up to 54 customers using LDAPS causing logins to the client to fail and therefore affecting screen sharing for users that weren’t already logged in.
A database server required an unplanned restart. This would have affected all customers causing issues with login attempts to both the admin portal and the client and therefore affecting screen sharing for users that weren’t already logged in.
Timeline
0927 AEST First support call was received with regards to LDAP authentication
0928 AEST Our monitoring system alerted us that 500 errors were being returned from our load balancer
0937 AEST After initial trouble shooting support team handed incident over to engineering team
0940 AEST Engineering team ascertained that load had been rising gradually on the database since 0840 and it was now at maximum
0951 AEST Engineering team identified the cause of the abnormal load and took measures to stop it
1000 AEST Engineering team restarted the database server to clear the queue
1005 AEST Database load returned to normal
1007 AEST 500 errors from the load balancer ceased and the issue with the admin portal and client logins was resolved
1010 AEST Customers still reporting issues with LDAP authentication
1023 AEST Engineering team identified that there was an incompatibility between our LDAP authentication and the new version of the software framework that was deployed that only occurred within a specific code path related to certain configurations
1151 AEST Engineering team identified the cause of the incompatibility
1200 AEST Engineering team implemented and started testing fix
1209 AEST Engineering team deployed fix and LDAP issue was resolved
1320 AEST During the post incident analysis the engineering team discovered that the abnormal database load had had an adverse effect on the database server and performance was degrading
1340 AEST The engineering team concluded that the degradation of the performance of the database server would continue until it was restarted
1404 AEST The engineering team made the decision to restart the database to restore the performance as it was starting to cause intermittent issues
1412 AEST The database restarted and performance was returned to normal
Root Cause
The abnormal load on the database server was caused by an increased number of connections and a doubling up of scheduled jobs, ironically caused by the by the old servers that had been retired which had been left on hot standby in case we encountered any issues with the new servers and enable us to fail back quickly.
A specific LDAP configuration caused a code path to be executed that was not in our test plans and therefore the bug was not identified in testing.
A database server required a unplanned restart due to the abnormal load that it was placed under during the initial incident.
Resolution and Recovery
0954 AEST The old application servers isolated from network
1000 AEST The database server restarted
1007 AEST Issues with the admin portal confirmed as resolved
1151 AEST Engineering team isolated issue to the net-ldap library and configuration affecting LDAPS
1200 AEST Engineering team identified configuration changes required to resolve issue
1209 AEST Engineering team deployed fix and issues with LDAP authentication was resolved.
1340 AEST The engineering team concluded that the database server needed to be restarted and configuration modified
1404 AEST The database server was restarted
1412 AEST The intermittent issues were resolved
Corrective and Preventative Measures
Increased monitoring added to the database server to ensure earlier notification of potential issues
Upgrade and migration procedure updated with learnings to ensure that there is not a repeat occurrence
Problematic LDAP configuration added to test plans to ensure that all code paths are tested
Increase test automation suite coverage
Review risk assessment process