We are happy to report that the increased processing power we implemented in the past week achieved the desired result and the transcoding queue is back to normal levels. We are closing this incident, and our engineering team will continue to monitor the transcoding performance, as we always do, with the extra processing power at our disposal to address any increase in traffic and demand for this service going forward.
Posted Mar 28, 2020 - 19:15 UTC
Update
We are continuing to monitor transcoding and upload jobs. Queues are still catching up to additional traffic. Based on the increased processing power from this week's capacity expansion, the current queue and new traffic, we can estimate that we will be back at normal processing time by early next week. We will continue to update until we return processing to normal performance.
Posted Mar 27, 2020 - 20:47 UTC
Update
We are continuing to monitor transcoding and upload jobs, following the most recent capacity expansion on March 25th. Queues are still catching up to the additional traffic that we're seeing globally as a result of Covid-19. We will continue to address job services and update here until we return processing to normal performance.
Posted Mar 26, 2020 - 22:30 UTC
Update
Due to increased demand for services, we have further increased our capacity, since our last update to you, and added additional servers to meet transcoding needs. As of today, processing capacity has been increased significantly. Our systems are still navigating the backlog of queued jobs, and we will continue to monitor and update on this status page until normal performance in transcoding jobs and uploads has been achieved.
Posted Mar 25, 2020 - 21:35 UTC
Update
In the next 24 hours, we will provide another update on queue status for upload/transcoding jobs. Our teams are continuing to monitor until queue backlog has returned to normal performance. If you need additional assistance with the status of pending jobs, please create a new case to Customer Care: https://support.kaltura.com
Posted Mar 24, 2020 - 19:02 UTC
Update
Infrastructure improvements to increase capacity for Transcoding and Upload jobs have been completed by our Operations team. These additional servers are processing the queue, both new jobs and pending jobs. In the next 24 hours, we will update on progress of restoring queues to normal performance. We will continue to monitor until this is completed, and going forward to ensure that performance has been optimized.
Posted Mar 23, 2020 - 17:48 UTC
Update
Earlier today, March 22nd, between 11:09am (UTC) to 11:19am (UTC) about 30% of our users may have experienced disruption in all services, which was the result of a wrong configuration that was implemented by mistake in one of our data centers (West Coast). The human error took place during an ad-hoc maintenance service, related to our mitigation plan for scaling up the service, to better handle the coronavirus hyper-load.
The issue was identified by our engineers and within few minutes services were resumed. Since then we have been monitoring the situation and we can confirm that the issue was resolved right away and since then the performance of both Data Centers has been normal. Furthermore, in the past few hours we already conducted a post-mortem and updated our guidelines to our teams to prevent such an event from happening again.
We are still monitoring and working diligently to resume normal performance around Transcoding. On that front performance has been stable, with the same status since our last report from yesterday, March 21st. The work that is underway, to improve on this issue, is still on track to be completed before 1pm (UTC). Updates will be posted here on this page.
Posted Mar 22, 2020 - 20:55 UTC
Update
During the last hour, customers might have experienced upload issues using FTP or Aspera. Currently, this problem has been resolved and our engineering team is continuing to monitor.
Posted Mar 22, 2020 - 12:32 UTC
Update
Kaltura is currently investigating an issue on West Coast DC. Traffic has been shifted to East Coast DC. More updates will follow on this page.
Posted Mar 22, 2020 - 11:27 UTC
Update
We are continuing to monitor the situation around Transcoding. Performance has been stable, with the same status since our last report. Our engineers have been working on this issue diligently, with actions being done continuously, including over the weekend, so we expect to be in the position to share more updates, here on this page, in 24 hours from this post
Posted Mar 21, 2020 - 11:49 UTC
Update
We are continuing to monitor the situation around Transcoding. Performance has been stable with the same status since our last report. We expect to be able to post further updates in the coming hours, and report improvements with the Transcoding performance, as our engineers diligently have been working on this issue.
Posted Mar 20, 2020 - 13:05 UTC
Update
We are continuing to monitor for any further issues.
Posted Mar 19, 2020 - 17:26 UTC
Update
We are experiencing delays in upload processing, which may present as longer times to upload entries or for entries to finish conversion. Root cause has been identified and mitigation steps are being taken by our engineering team. Services will be improved within the next few days; our team is completing these steps as a top priority. Please contact Customer Care with additional questions: https://support.kaltura.com
Posted Mar 19, 2020 - 17:02 UTC
Monitoring
Services have been restored. Our engineering team was alerted to service alerts for API, KMS and Uploads and functionality had a temporary degradation for ~1 hour timeframe. Our engineering team mitigated the issue, and services have been restored to normal functionality.
Posted Mar 19, 2020 - 14:09 UTC
Update
We are continuing to investigate this issue.
Posted Mar 19, 2020 - 13:59 UTC
Investigating
We are investigating alerts on API.
Posted Mar 19, 2020 - 13:45 UTC
This incident affected: Kaltura API, Transcoding, and Kaltura MediaSpace.