streamroot / fmse Goto Github PK

During the decoupling of this polyfill from our player, we removed a feedback loop that would get the key frame time from the JS dash player, and then:

1. do the actual seeking on a key frame
2. while making sure everything went well if audio and video segments were not aligned (basically making sure we didn't append an audio segment that started before the chosen key frame).
   Failing to ensure those 2 points would cause NetStream to crash in the weirdest ways.

A better way to work around these limitations is to give every frame located before the seek target the timestamp of the seek target itself.

This can be done in the transmuxer: this loop goes over every frame to write them in a ByteArray

I don't think we broke anything else during the decoupling phase, but a good way to start might be to seek programmatically exactly to the beginning of a DASH segment (expecting that it starts with a key frame), using a video only stream.

In appendBuffer call, as of now, we still need to pass the segment's start and end timestamp, and wether it's an initialization segment. See here
This is the main thing that still require some modifications in the JS player to work with this polyfill.

We need to parse these infos from the MP4 segments themselves in order to get rid of this issue

It's important to define from the start what our target browsers will be, to have a more precise idea of the scope of our polyfill and the workarounds we are going to implement.

If we want to support IE 9- for example, then we want to have a satisfying degraded version without web workers and that would solve #1 . If we don't care about Safari at all since MSE is in the latest versions, then we can close #1 already

In Safari, btoa is not supported in web workers, but we make use of it here:
https://github.com/streamroot/fmse/blob/master/src/js/lib/B64Worker.js#L22

Dash.js, and probably others, test if the video is an instance of HTMLMediaElement : https://github.com/Dash-Industry-Forum/dash.js/blob/development/src/streaming/utils/Capabilities.js#L66

We used to inherit HTMLMediaElement prototype to work around this, but it broke when updating the version (Error: illegal invocation when setting video.src): https://github.com/Dash-Industry-Forum/dash.js/blob/ae5330eb3a6112fda9f4b831b2620f4a58990b44/src/streaming/models/VideoModel.js#L88

We don't have anything like the controls given by <video controls>...</video>.

I don't think it is useful for anybody in production, but on the other hand it would be great to have a reusable standalone js tool that would add basic control to any sample page we would build for other MSE based player

in my player just if mediasource.sourceopen is called the player is visible.
but here if player is hidden sourceopen is not executed.
I m investigating if it simple to modify the code

dash.js scheduler sometimes stops downloading segments, sometimes resumes after a while.
This doesn't happen on regular MSE, so I suspect some part of the API (getter or event) is not implemented correctly.

Very easily reproductible on http://wowza.streamroot.io/vodorigin/tos.mp4/manifest.mpd .

If you reconfigure dash.js buffer to download 15s of content, it stops after downloading 3 segments:

player.setStableBufferTime(15);
player.setBufferTimeAtTopQuality(15);
player.setBufferTimeAtTopQualityLongForm(15);

I can t call video.volume

I think this is a great idea to provide a fallback for Media Source Extensions. Thanks for putting this concept out there!

I am interested in why we need the Service Worker dependency?

If you look at browser support for MSE and the browser support for Service Workers I'm not sure what browsers this is beneficial for. Are there any browsers that don't have MSE that support Service Workers?

Since the service worker requirement makes this apply to such a small subset of browsers, what is the goal of this library?

hi ,
your project it is very interesting i m working for integrating it in my player :)

I emproved a bit the init call:

• style is copied by video el
• it is possible to create also if video element is not in the dom
• parent node is more generic

function init(onReady, options) {
	if (!options) options={}
	var polyfillSwfUrl=options.polyfillSwfUrl || 'fMSE.swf';
	var flashByDefault=options.flashByDefault || false;
	var videoElement=options.videoElement||null;
	var parent=options.videoContainer||(videoElement&&videoElement.parentElement)||document.body;
    var isMSESupported = !!window.MediaSource;
    if (isMSESupported && !flashByDefault) {
        return onReady(videoElement);
    }
    window.fMSE.debug=options.debug||false;
    

    window.MediaSource = MediaSourceFlash;

    window.fMSE.callbacks = window.fMSE.callbacks || {};
    window.fMSE.callbacks.onFlashReady = function () {
        onReady(new VideoExtension(swfObj));
    };

    var readyFunctionString = "window.fMSE.callbacks.onFlashReady";

    var height = options.height||(videoElement&&videoElement.height) || 150;
    var width = options.width || (videoElement&&videoElement.width) || 300;
    
    var oldId , oldIdClasses ;
    if (videoElement){
		  oldId = videoElement.id;
         oldIdClasses = videoElement.className;
		
	}
	
	var swfObj = document.createElement("object");
    oldId='e'
    if (oldId) {
		swfObj.setAttribute('id',oldId); 
		swfObj.setAttribute('name',oldId) 
    }
   swfObj.setAttribute('type','application/x-shockwave-flash')
   swfObj.setAttribute('data',polyfillSwfUrl)
   swfObj.setAttribute('width',width)
   swfObj.setAttribute('height',height)
   
    if (oldIdClasses)swfObj.setAttribute('class',oldIdClasses)
   swfObj.setAttribute('style','display:'+(videoElement &&videoElement.style.display?videoElement.style.display:'block'))
    var param = document.createElement("param");  
    param.setAttribute('name','movie')
    param.setAttribute('value',polyfillSwfUrl)
    swfObj.appendChild(param);
     param = document.createElement("param");  
    param.setAttribute('name','flashvars')
    param.setAttribute('value',"readyFunction="+readyFunctionString)
    swfObj.appendChild(param);
     param = document.createElement("param");  
    param.setAttribute('name','allowScriptAccess')
    param.setAttribute('value','always')
    swfObj.appendChild(param);
    
       param = document.createElement("param");  
    param.setAttribute('name','wmode')
    param.setAttribute('value','opaque')
    swfObj.appendChild(param);
    
        param = document.createElement("param");  
    param.setAttribute('name','allowNetworking')
    param.setAttribute('value','all')
    swfObj.appendChild(param);
       param = document.createElement("param");  
    param.setAttribute('name','bgcolor')
    param.setAttribute('value','#000000')
    swfObj.appendChild(param);
    parent.appendChild(swfObj);

    
}

Considering the browsers for the future wants support just mp4 and webm it would be usefull fmse support also trascoding for webm format to flash.

Seek adds a lot of complexity in the appending/transmuxing pipeline: NetStream has limitations and will crash quite spectacularly if you append segments out of order.
That's why we need to make sure every segment is properly flushed, at every step of the pipeline, and we added a lot of state variables along the way to handle situations like: "do I need to discard this segment when it comes out of the worker?"
It would be great to improve the SourceBuffer.as class to handle segments out of order, from disjoint time intervals.

We would need to account for #4 : if we don't be careful about this some segments could have their first frames squashed if they were appended during a seek

This is probably a wontfix

We're far from covering 100% of the specification of Video, MediaSource, and SourceBuffer APIs.

This is meant as a tracking bug to centralize this info although we could most probably find a way better tool to discuss, list and prioritize the API details we're missing.