Resolved -
What Was the Impact: The majority of customers have verified normal performance was restored earlier today at 9:20 p.m. UTC.
Please note that DNS changes may take up to 24 hours to propagate, which means that some content rendering may return to normal by or before 9:20 p.m. UTC on May 15.
Current Status: Testing and monitoring have been completed.
What to Expect Next: The incident is now closed.
May 14, 22:55 UTC
Monitoring -
What Was the Impact: Due to an issue at a CDN vendor, there were issues with content, which also affected playback, player and thumbnails.
Current Status: As of 9:20 p.m. UTC normal performance has been restored.
After a fix was made by the vendor, we have completed removing the earlier DNS workarounds. Testing has also been completed.
What to Expect Next: We are currently monitoring the system.
May 14, 21:27 UTC
Update -
What is the Impact:
Some customers may still be seeing issues with thumbnails, and some customers may still see issues with content, however, content for the majority of customers was restored earlier through mitigation steps. However, we are making additional changes to restore full performance.
Current Status:
Earlier today we implemented workarounds to mitigate impact from an issue at an external vendor for CDN/content. These workarounds included DNS changes, which can take some time to propagate, which is why some customers may still be seeing issues with thumbnails and content display.
Within the hour, our external vendor has completed a necessary change. Our next steps will be remove the earlier workarounds to return to normal performance.
What to Expect Next:
We will update after workarounds have been removed and testing completed, or within 1 hour.
May 14, 20:42 UTC
Update -
What is the Impact: Playback/content was previously restored for the majority of customers.
- A limited number of customers using tokens for CDN may still experience issues with availability of content.
- Some customers may see that thumbnails have not yet populated in their account -- thumbnails will populate as soon as DNS changes finish propagating.
Current Status: We are in the process of completing additional mitigation steps in collaboration with an external vendor.
We are seeing performance restored for the majority of customers, however, we are working to address remaining items as quickly as possible, specifically for customers using tokens for CDN.
What to Expect Next: The next update will be posted within 1 hour.
May 14, 19:51 UTC
Update -
What is the Impact: Playback/content is now restored for the majority of customers.
Customers using tokens may still experience issues with availability of content.
Current Status: Additional mitigation steps have been completed in collaboration with external vendors. We are seeing performance restored for the majority of customers, however, we are working to address remaining items as quickly as possible, specifically for customers using tokens.
What to Expect Next: The next update will be posted within 30 minutes.
May 14, 18:37 UTC
Update -
What is the Impact: Inconsistent availability of content for playback and availability of player.
Current Status: Our engineering team has been collaborating with external vendors on resolution and additional mitigation steps. Our teams continue to work to resolve open issues.
What to Expect Next: The next update will be posted within 30 minutes.
May 14, 18:21 UTC
Update -
What is the Impact: We are currently working to resolve issues related to content/playback.
Although performance was improved for KMC, KMS and player, we are still working toward full resolution for these and other open issues as our top priority.
Current Status: Our engineering team continues to work as quickly as possible to resolve open items.
What to Expect Next: The next update will be posted within 30 minutes.
May 14, 17:37 UTC
Update -
What is the Impact: We are currently investigating issues identified during testing, impacting KMC and Thumbnails.
Current Status: Our engineering team continues to work to resolve open items.
What to Expect Next: The next update will be posted within 30 minutes.
May 14, 17:08 UTC
Identified -
What was the Impact: Entries were not playing back. Player were external pages not loading.
Current Status: Our teams have mitigated the issue and are currently testing performance.
What to Expect Next: The next update will be posted within 30 minutes.
May 14, 16:51 UTC
Investigating -
What is the Impact: Entries are currently not playing back. Player on external pages is not loading.
Current Status: Our teams are currently investigating.
What to Expect Next: The next update will be posted within 30 minutes.
May 14, 16:35 UTC