Issues with call recording
Incident Report for Gong
Postmortem

On Apr. 12 at ~9am PST, we received an internal alert indicating that the web conference recording subsystem may be malfunctioning.

We began an investigation, and the issue was traced to a significant load on our primary production database. It appeared like the high load was due to an extremely large number of processing machines pulling tasks from the queue to retroactively apply new Gong functionality to historical calls.

Once the issue has been identified, the task queue was emptied and the database server was restarted. Subsequently, recording proceeded normally.

On Apr. 15, we've conducted a postmortem analysis, yielding several action items to reduce likelihood of similar issues in the future:

  • The task queuing mechanism is being re-engineered to avoid being affected by a large number of processing machines.

  • The recording subsystem is being migrated to a separate database, to avoid potential adverse impact by extraneous load in other subsystems.

  • A subset of the data stored in our primary database, identified as a potential mid-term bottleneck, will also be migrated shortly into a separate data store.

Again, we apologize for any inconvenience this incident may have caused.

Posted 8 months ago. Apr 16, 2018 - 19:02 UTC

Resolved
The issue is confirmed to have been resolved as of lately.

We will keep you posted as we complete the root cause analysis and devise an action plan to avoid similar issues in the future.

Again, we apologize for any inconvenience.
Posted 8 months ago. Apr 12, 2018 - 19:43 UTC
Identified
We've identified the root cause behind the web conference recording issue, and are working to resolve the issue.
Posted 8 months ago. Apr 12, 2018 - 18:32 UTC
Investigating
We've encountered some issues with recording of web conferences. We are investigating the issue and will update as soon as we've identified the underlying issues. Processing of calls made through phone systems is functioning normally.

We apologize for any inconvenience while we look into this issue.
Posted 8 months ago. Apr 12, 2018 - 17:02 UTC