Current Status: The August 26th incident that affected API, VOD, and KMS/KAF was resolved yesterday at 5:38 pm UTC. Content delays were resolved at 22:18 UTC yesterday.
What to expect next: Engineering is taking steps to prevent the issue from occurring again. This incident is now closed.
Full details will be provided in the Root Cause Analysis document (RCA) which will be available within 7 business days. If you created a case with Customer Care yesterday, you will be sent the RCA when published -- so no need to reach out to again. Otherwise, please create a new case with CC to request the RCA.
Posted Aug 27, 2020 - 14:43 UTC
Update
We are continuing to monitor for any further issues.
Posted Aug 27, 2020 - 11:38 UTC
Update
What was the impact: Delays on content visibility in KMS and/or errors accessing some content in KAF should now be resolved.
Current Status: Earlier today, engineering team identified delays on content visibility in KMS. This was a symptom of the earlier issue. Our engineering team has now resolved content delays or errors.
What to expect next: Our engineering team will continue to monitor overnight. The next update will be after monitoring on Thursday, August 27th.
Posted Aug 26, 2020 - 22:18 UTC
Update
What is the impact: Delays on content visibility in KMS and/or errors accessing some content in KAF. Recently uploaded entries or new playlist entries may not be immediately visible.
Current Status: During monitoring , our engineering team identified delays on content visibility in KMS. This is a symptom of the earlier issue. The delays are within eSearch (secondary data layer). Our engineering team is working to resolve content delays or errors.
What to expect next: The next update will be posted within 1 hour.
Posted Aug 26, 2020 - 20:31 UTC
Update
What is the impact: Delays on content visibility in KMS. Recently uploaded entries may not be immediately visible.
Current Status: During monitoring of the services (API/KMS) that were restored at 5:38pm UTC, our engineering team has identified delays on content visibility in KMS. This is a symptom of the earlier issue. The delays are within eSearch (secondary data layer), and no current issues in the database. Our engineering team is working to resolve delays for content.
What to expect next: Content delays are expected to be cleared soon. The next update will be posted within 1 hour.
Posted Aug 26, 2020 - 19:19 UTC
Update
What was the impact: API, VOD, and KMS/KAF functionality
Current Status: Services listed above were fully restored as of 5:38pm UTC. The incident was caused by an issue within the current database infrastructure. It was not related to current migration efforts.
What to expect next: Our engineering team will monitor closely for the remainder of the day to ensure normal performance. Engineering is taking steps to prevent the issue from occurring again.
Full details will be provided in the Root Cause Analysis document (RCA) which will be available within 7 business days. If you created a case with Customer Care earlier, you will be sent the RCA when published -- so no need to reach out to again. Otherwise, please create a new case with CC to request the RCA.
Posted Aug 26, 2020 - 18:24 UTC
Monitoring
What is the impact: API, VOD, and KMS/KAF functionality have been restored.
Current Status: Issues affecting the platform have been resolved. Normal functionality is restored to API, VOD, KMS/KAF.
What to expect next: Our engineering team will monitor closely for the remainder of the day to ensure normal performance.
Posted Aug 26, 2020 - 17:50 UTC
Update
What is the impact: API errors, VOD playback issues, intermittent access to KMS/KAF.
Current Status: Our teams are working to restore normal performance as the top priority with all available resources.
What to expect next: The next update will be posted as soon as available or within 1 hour.
Posted Aug 26, 2020 - 17:16 UTC
Update
What is the impact: API errors, VOD playback issues, intermittent access to KMS/KAF with errors.
Current Status: Our engineering team is working to mitigate and resolve all application errors and restore normal performance as the top priority.
What to expect next: The next update will be posted as soon as available or within 30 minutes.
Posted Aug 26, 2020 - 16:37 UTC
Update
What is the impact: API errors, VOD playback issues. KMS/KAF is now intermittently accessible with the slower performance for some accounts.
Current Status: Intermittent access for some accounts is available for KMS/KAF; we are working to restore full access and functionality. Our engineering team is working to mitigate and to resolve all application errors as the top priority.
What to expect next: The next update will be posted as soon as available or within 30 minutes.
Posted Aug 26, 2020 - 15:59 UTC
Update
What is the impact: API errors, VOD playback issues. KMS/KAF is now accessible with slower performance.
Current Status: Traffic was shifted at 2:33 pm and 2:43 pm UTC but did not mitigate the current issues. Our Engineering team has restored access to KMS/KAF. They are taking steps to resolve and continues to work to restore normal performance as the top priority.
What to expect next: The next update will be posted as soon as available or within 30 minutes.
Posted Aug 26, 2020 - 15:29 UTC
Update
What is the impact: API errors, KMS / KAF access unavailable, VOD playback issues
Current Status: Our engineering team shifted traffic at 2:33 pm and 2:43 pm UTC as part of mitigation steps. Our team continues to work to restore normal performance as quickly as possible.
What to expect next: The next update will be posted as soon as available or within 30 minutes.
Posted Aug 26, 2020 - 14:56 UTC
Update
We are continuing to investigate this issue.
Posted Aug 26, 2020 - 14:33 UTC
Investigating
We are currently investigating cross-component issues affecting the platform. Our team is working to resolve it as quickly as possible.
Posted Aug 26, 2020 - 14:32 UTC
This incident affected: Kaltura API, Playback, and Kaltura MediaSpace.