Versions Compared

Old Version 3

changes.mady.by.user Patrick H

Saved on

compared with

New Version 4

changes.mady.by.user Patrick H

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

As noted, some Vivi app users also experienced issues, particularly those attempting to log in on a device for the first time. Based on our data and the timing of the outage, we estimate that approximately 11% of our North American customers may have been affected during this period.

Timeline

All times below are in AEST.

  • Midnight, 30th October 2024 (Melbourne Time) / 9 AM EST (Previous day, 29th October 2024 (EST)

  • 1:16 AM (AEST) / 10:16 AM (EST, 29th October) - First customer ticket raised.

  • 1:20 - 1:47 AM (AEST) / 10:20 - 10:47 AM (EST, 29th October) - Support engineers begin investigating and conduct standard troubleshooting.

  • 1:48 AM (AEST) / 10:48 AM (EST, 29th October) - Issue escalated to on-call engineers.

  • 1:57 AM (AEST) / 10:57 AM (EST, 29th October) - First on-call engineer goes online.

  • 2:51 AM (AEST) / 11:51 AM (EST, 29th October) - Initial troubleshooting deemed ineffective; heartbeat traffic reduced to ease database load. Further actions deferred until 6 AM for additional engineer support.

  • 6:00 AM (AEST) / 3:00 PM (EST, 29th October) - Two additional engineers come online.

  • 6:47 AM (AEST) / 3:47 PM (EST, 29th October) - Third additional engineer joins.

  • 6:50 - 6:57 AM (AEST) / 3:50 - 3:57 PM (EST, 29th October) - Decision to sequentially restart backend services and further reduce non-critical service traffic.

  • 7:34 AM (AEST) / 4:34 PM (EST, 29th October) - Initial issue marked as resolved. Heartbeat degradation anticipated due to redirected traffic.

  • 9:52 - 10:50 AM (AEST) / 6:52 - 7:50 PM (EST, 29th October) - Heartbeat traffic restored to 100%.

  • 11:00 AM (AEST) / 8:00 PM (EST, 29th October) - Resolution of heartbeat issues communicated to customers; major incident resolved.

...