Leading up to 8:15PM (GMT +10) our host discovered some issues in their database clusters. This lead to connections from Attendly and other customers not being dropped from the database when they should be.
At its peak, connections from Attendly were 8x the normal range. This caused our host to suspend the database (block it). At this point the Attendly platform responded with timeout errors (connections to the database were timing out).
Registration pages and the admin portal showed gateway timeout errors to any user who tried to access them.
After discovering that the issue lay at the database level, we contacted our host and they confirmed the database had a block on it. They were able to remove the block, but once it was removed the issue began to occur again. We had to stop all Attendly services until the database had normalised, before gradually adding services back.
We apologise for the downtime and are working hard to make sure that this does not happen again.
Posted about 2 years ago. May 24, 2016 - 13:43 AEST