Remote Playback API
Home Page: https://www.w3.org/TR/remote-playback/
License: Other
This repository contains the Remote Playback API specification that is being worked on in the W3C Second Screen Working Group.
Please refer to the group's Work Mode for instructions on how to contribute.
Notes in specs are by nature informative and as such should not contain normative statements. Notes in the current draft of the Remote Playback API that contain normative statements should either be adjusted to remain informative, or the content should be turned into real normative statements, to remove all ambiguity on the informative/normative nature of the text.
Notes that need some update:
This is a meta issue to keep us aware of the fact that we should attempt to publish a First Public Working Draft (FPWD) of the spec in the near future. FPWD is the first step in the formal Recommendation Track.
To be FPWD ready, we should add some prose to complement the IDL (basically, formalise the draft summaries a bit, no need to be perfect).
@mounirlamouri @avayvod It'd be great if you could triage the open issues and label the ones you feel would benefit the most from the F2F discussion. We could try to publish the FPWD already before the F2F to give it more visibility and attract outside feedback.
If there is a remote playback happening and the device gets disconnected, the state would switch to disconnected and likely the availability callback will be called. Should we define the order? or is it already define?
The spec currently reads,
If the disableRemotePlayback attribute is present on the media element, the user agent must not play the media remotely or present any UI to do so.
But the media element must still return an instance of RemotePlayback. The behavior of this object should be defined when disableRemotePlayback is present.
Tracking issue for the following problem:
Following the Presentation API experience, the page would get an id from the start() call / onconnected event and then pass it to another call like join(playbackId) to attach the media element to the remote playback session without prompting the user again.
Currently the behavior when a page adds the disableRemotePlayback attribute to a media element that's being played remotely is undefined. Options are:
Option 2 seems like the most reasonable to me.
Step 3 says that OperationError should be thrown when trying to connect while already trying to connect somewhere else. I assume the rationale is that UA might end up showing two prompts on top of each other. However, this is a pure UI consideration and it seems that making this not a requirement for proper implementation is important: there might be some type of UI design that would make things work fine. For example, in-page UI anchored to the element as opposed to modal UI.
According to the specification, calling cancelWatchAvailability will reject if disableRemotePlayback is true. However, setting disableRemotePlayback is not expect to stop watch which means that we might end up in situations where per spec, the callback to watch availability will have to run for ever.
We should either ask that disableRemotePlayback cancels watch or that cancelWatchAvailability doesn't reject if disableRemotePlayback.
In my opinion, we should do the former to stay consist with the spirit of the specification: when disableRemotePlayback is set to true, UA is expected to stop remote session. It should also stop watching for availability.
From #83 (comment)
@avayvod proposes to fire availability callbacks with false before clearing the global set of callbacks, so that pages do not continue to believe that there are available playback devices.
Before we request horizontal reviews on security and privacy, the group should review and assess the Remote Playback API against the Self-Review Questionnaire on Security and Privacy, and suggests text for a possible security and privacy section in the spec, similar to the one in the Presentation API.
The Fingerprinting Guidance for Web Specification Authors and Privacy Considerations for Web Protocols may prove useful documents to review as well.
Anyone willing to do that?
The step 8 from the prompt() algorithm was inspired by the similar step (10.2) in the Presentation API's PresentationRequest.start() algorithm.
This step seems problematic to implement when the availability monitoring is not supported by the device. At least on Android the device picker dialog will not hide until user closes it even if devices are not found. Closing the dialog by the browser would be against the current guidelines of using it I assume.
I propose making this step optional.
@mfoltzgoogle - I'm not sure how this works with the Presentation API on desktop (I think on Android it is not supported as well).
In preparation for the wide review, it would be good to update the SoTD section, clarify the "This document is a work in progress and is subject to change" note to better reflect reality knowing implementations are coming together. Also, note the status of existing issues. My assessment says the remaining open issues are considered minor, except issue #41 that suggests extensions to HTMLMediaElement and HTMLVideoElement interfaces; would be good to note this explicitly to attract further feedback and suggestions during wide review. Let me know if you feel the same.
The reason for updating SoTD is simply to make it easier for people outside this group reading the wide review spec to be informed of the stability and completeness of the spec, and give better feedback.
@tidoust volunteered to address this issue. Thanks!
The remote playback device capabilities might not be always a subset of those of the local playback device. The cases when something is not working locally but can work remotely might be worth looking into and adding a note about too.
Should research and document such capabilities to see how to best reflect that in the spec.
I think the use of a separate Availability object for remote playback device for this use case causes some problems for the spec.
One alternative would be to have an enum + event to track availability changes on the remote object itself, which would seem to address the issues above. Availability would be kept in sync with both device availability and the media source list.
partial interface RemotePlayback : EventTarget {
readonly attribute RemotePlaybackAvailability availability;
attribute EventHandler onavailabilitychange;
enum RemotePlaybackAvailability {
"available",
"unavailable",
"unknown" // Background monitoring not supported.
};
};
https://w3c.github.io/remote-playback/#extension-to-the-htmlmediaelement
If it always returns the same object, [SameObject] can be used.
Currently, the spec doesn't define how the remote playback is stopped. It could be some user agent provided UI that the user can trigger somehow or that can be shown when the page calls start() on the remote instance that's already in a connected state. However, it seems natural to allow the website to have a custom control on the media element that stops the remote playback.
partial interface RemotePlayback {
Promise stop();
}
Since the media element can have more than one source, the UA should be able to remote what it thinks is the best for the selected remote device.
Also maybe allow the website to specify the preferred source for remote playback (e.g. on mobile the element may have the default local playback source in SD quality while a link to an HD media could be provided for remote playback specifically).
The website could already just change the element source when it's being played remotely though.
6.2.1.3 Getting the remote playback devices availability information
https://w3c.github.io/remote-playback/#getting-the-remote-playback-devices-availability-information
Would there be a need to query remote device playback availability based upon language and our country? Could we have a situation where there would be some specific hardware that would need to be selected based upon language? For example, could there be a case where the remote playback device had a specialized set of script specific fonts that were going to be used to display captions.
Both steps essentially mean the case when the current source of the media element is not compatible with found devices (although step 5 doesn't require running the monitoring algorithm since it covers the case of fundamentally unsupported sources while step 9 is for the case when the source is only unsupported by the devices found).
I think it's confusing and merging 5 into 9 would probably be easier to understand.
@mfoltzgoogle @mounirlamouri WDYT?
We need implementations to be able to tell the page that something can't be played remotely because it is not supported by the other end. It can be because of EME, because of audio/video codecs or maybe because of MSE.
I did not think yet of the mechanism but I think having connect() reject with some error (like NotSupported) could be the right approach.
To be able to publish a Candidate Recommendation #73, we need a preliminary implementation report similarly to the Presentation API test results (see test results column).
@avayvod @mounirlamouri, are you happy with the current coverage of the Remote Playback API test suite? (You added idlharness tests that were initially missing, is TODO up to date?)
@Honry could perhaps help you produce a preliminary implementation report using the Chrome for Android implementation https://crbug.com/578833 if you so prefer, and/or help update the test suite if some tests are still missing.
Go to https://w3c.github.io/remote-playback/#remoteplayback-interface you will see that you can click most items in the interface apart from cancelWatchAvailability
A spin-off from the issue #3:
state is "disconnected" or "connecting", all interaction with HTMLMediaElement is forwarded to/from the local playerstate is "connected", all interaction with HTMLMediaElement is forwarded to/from the remote playerstate changes from "connecting" to "connected", the last known state of the local player is passed to the remote playerstate changes from "connected" to "disconnected", last known state of the remote player is passed to the local player@foolip are there any methods we'd need to disable in "connecting" state? (for example, pause() is allowed but play() could be rejected)?
IMO, "6.2.2 Prompt user for changing remote playback state" should be changed to include a step between current 11 and 12 so that if the state was connected and remained connected. Such that the user did not change the connecting device, the promise should be fulfilled. At the moment, it would be rejected which could be very surprising when the user is in a connected state because the UA could include options in this dialog such as changing volume stream, etc. Rejecting the promise would mostly create noise for the website.
It looks like setupRemotePlayback() is called twice when the user clicks remotePlaybackBtn, once in the resolver for connect() and again in the onstatechange event handler when the state transitions to connecting.
Also, you may want to disable remotePlaybackBtn when the state changes to connecting, and re-enable it on a transition to disconnected. Maybe this would be done in the setup/stop helper functions?
https://lists.w3.org/Archives/Public/public-privacy/2017JanMar/0009.html
- In the section “Disabling remote playback”, consider to add the requirement that the monitoring of devices must not occur if the feature is disabled by the user, thus “If the disableRemotePlayback attribute is present on the media element, the user agent MUST NOT monitor availability, play the media remotely or present any UI to do so.
Should update steps to ensure that monitoring is stopped when the set of availability callbacks has been emptied?
The page might want to know when the remote state changes to show different UI or change the source appropriately.
The controls attribute will have the UA render default media controls for the media element [1]. If a media element with controls set is remoted, the spec might want to note that the default controls should be updated accordingly for the remote case (i.e. the attribute is still respected even in remote mode), and possibly give an implementation suggestion like using a poster image in place of the rendered <video>.
[1] https://www.w3.org/TR/html5/embedded-content-0.html#attr-media-controls
This issue is related to Issue #46.
I didn't see any language in the spec that the user agent initiating remote playback is required to stop rendering media in the element after connection to a remote playback device. But I would assume this behavior is intended, unless there is a use case for simultaneous playback (i.e., connection to a recording device or showing the same media in two rooms).
This meta issue tracks progress toward the Remote Playback API Candidate Recommendation (CR) publication. Editor: @mounirlamouri
By publishing a CR we:
The CR requirements are the following from the W3C Process perspective:
must show that the specification has met all Working Group requirements, or explain why the requirements have changed or been deferred,
must document changes to dependencies during the development of the specification,
must document how adequate implementation experience will be demonstrated,
must specify the deadline for comments, which must be at least four weeks after publication, and should be longer for complex documents,
must show that the specification has received wide review,
may identify features in the document as "at risk". These features may be removed before advancement to Proposed Recommendation without a requirement to publish a new Candidate Recommendation.
It is good practice to evaluate the spec against internationalization techniques before requesting horizontal review from the i18n group. Most of these points do not seem to apply to the Remote Playback API though.
It seems useful to take a look at the Presentation API and draw parallels. I think the following i18n-related aspects of the Presentation API may warrant notes in the Remote Playback API spec:
Some user agents may or may not be able to remote video with media keys set before or during remoting. The spec should provide some guidance as to what developers should expect when setMediaKeys() [1] is called on a remotable media element.
[1] https://w3c.github.io/encrypted-media/#htmlmediaelement-extensions
From #83 (comment)
@avayvod: I think this example implies the callback ids must be unique across all the media elements within a browser session which is not true. They only must be unique for each media element alone. An example is a counter in the media element's scope, that's set to 0 when the media element is created and is incremented each time its value is returned by watchAvailability() for this particular element as the callback id.
@mfoltzgoogle: That seems inconvenient if a page is watching availability on multiple elements and wants to keep the callbacks in a Map or otherwise track the set of callbacks by id.
@avayvod: Fair point. It might be more likely having one callback per many elements compared to many callbacks for a single element. The spec doesn't guarantee any cross-element uniqueness of the ids at the moment though.
I think there's an assumption that remote.state must be disconnected upon starting the "Initiate remote playback" algorithm, but there's no explicit check for it in the steps.
The algorithm defines the successful case but not the failure case; from the rest of the spec, it sounds like a connecting to disconnected state change + event should be fired at remote (versus connected for the successful outcome).
https://lists.w3.org/Archives/Public/public-privacy/2017JanMar/0009.html
- Does the RemotePlaybackAvailabilityCallback object include a human-readable name to identify the Callback object? E.g. “kitchen speaker”, “bedroom TV”, “Medical device”? Would this also be exposed outside of the UA? Privacy implications would vary depending on where this human-readable name is disclosed, if any.
It seems clear enough this is not the case, but perhaps we can add a sentence in the Privacy section clarifying this.
Import comment(#87 (comment)) from @mounirlamouri:
The issue with this API is that you will have to use a Chrome Cast compatible device to test the API on Chrome but another browser might not support Chrome Cast. The test preconditions will have to start with something a bit hand-wavy like "Have a device compatible with the browser on your WiFi". Because only Chrome (maybe other Chrome-based browsers) has an implementation for this at the moment, it would be hard to be clearer.
The Remote Playback API has different method names for actions that are conceptually very similar to the Presentation API.
For example, the Remote Playback API has watchAvailability(), which takes a callback parameter, whereas the Presentation API has getAvailability() that returns a PresentationAvailability object with an onchange event handler. Also, start() in the Presentation API is prompt() in the Remote Playback API.
Should we try to align these, for consistency? I think this may help with use of these APIs by application developers.
In remoting mode (i.e. state == connected) any side effects on the media element, for example mutations to properties, invocations of methods, or detachment from the DOM may (or may not) affect remote playback.
Because the behavior of the remote playback device seems to be out of scope for this spec, there may not be much to say in the normative sections of the spec.
However, in my opinion it would be a better spec to at least say something in regards to what should happen. I can see these behaviors falling into three categories:
The challenge will be in cases where the observable state of the element might be affected by implementation choices. For example, when playing back on a remote device that does not support changing the playback volume, how should the element behave when its volume attribute is set?
https://lists.w3.org/Archives/Public/public-privacy/2017JanMar/0009.html
- It is unclear if the callbackId is derived from a unique identifier on the Callback device (e.g. a hash value of a MAC address). Is there any reason why this could not be generated for each session by the UA? It would still be unique across all callback devices on the network but different devices on the same network could have a different set of unique devices and thus reducing the potential for device fingerprinting.
I think the spec is clear that it's unique to the media element. Should an algorithm be defined to generate it?
... and reject when user cancels the selection.
Having to wait for the Promise to be resolved and then to check if its value is true seems to make the examples less straightforward:
videoElem.remote.connect().then(switchToRemoteUI);vs.
videoElem.remote.connect().then(function(success) { if (success) switchToRemoteUI(); });In the Presentation API, we converted the API from having one statechange event to distinct events for each state, to improve developer ergonomics: fewer if/then/else blocks in a monolithic event handler.
(I can't find the specific GH issue where this was discussed, since I think it was an offshoot of other API discussions. Can dig some more if interested.)
Perhaps the Remote Playback API should do the same:
partial interface RemotePlayback : EventTarget {
attribute EventHandler onconnect;
attribute EventHandler ondisconnect;
};
However, the initial value of remote.state does not seem to be defined; that would need to be addressed to determine whether an additional connecting event is necessary.
Step 1 says that UA should reject with NotSupportedError if the UA knows the formats are not supported (so it doesn't event try to discover).
Step 8 says that the UA should reject with NotFoundError if no devices were found or all found devices are not compatible.
I wonder if we could do:
NotSupportedError when no compatible devicesNotFoundError when no devicesIt would require to reject different errors in substeps 1 and 2 from step 8.
This issue is filed so that everyone interested in the Remote Playback API is aware of the issue. It's not yet entirely clear which spec will have to make adaptions, once that is clear one of the issues can be closed.
This is a follow-up from the TAG review request.
I haven't done a particularly good job at getting back to the group with the feedback we had, I apologize for that. The API side of the spec is sound, and I don't see anything that would be a serious issue worth opening issues for - but the elephant in the room is interoperability.
Setting aside the fact that this is already being implemented and shipping, with each implementation possibly shipping using some form of proprietary transport - we think that the spec not defining a common denominator transport mechanism in a normative section is problematic and would hurt interoperability, and potentially jeopardize the usefulness of the API.
If implementations ship with incompatible transports, we will be stuck in a situation where content authors have to rely on sniffing and whitelisting, either in the form of code or documentation, which is undesirable - and this encourages tight coupling between the remote playback device and the user agent.
From what I see, with the spec as of today: (Correct me if I am wrong)
This doesn't seem like the kind of usability we should be promoting. I've brought this up with the team, and @w3ctag does not consider this specification ready for implementation for these reasons.
I have been made aware of the open screen protocol, would it be possible to accelerate the development of that and include that as a common denominator requirement for interoperability before we end up in another sniffing/whitelisting situation?
E.g. onconnected, ondisconnected, etc.
Having an event handler makes sense so that the web developer doesn't have to poll the attribute.
Per-state events vs a single onstatechange mostly for consistency with the Presentation API.
Represents the state between the time when a remote playback device has been picked by the user and when the playback of the media is ready to start on the device.
The justifications:
disconnected to connectedThe draft summary of state changes:
disconnectedconnecting; while in this state all media commands on the video element (play, pause, seek, volume, etc) throw exceptionsconnecteddisconnectedIt's not a very clean API, but i'm not sure that it's worth trying to deprecate existing content in order to get a cleaned up API. Previous similar attempts for other APIs have failed.
Step 8-1 of the getting availability algorithm [1] queues a task to invoke the callback with the availability status, but it's not clear what the value would be if the "monitoring a list" algorithm was not already running.
As a corollary, the "monitoring a list" algorithm does not need to be started again in step 8-2 if it was already running. Or is it assumed to run continuously in the background?
Also, in step 2-2 of "monitoring a list", the callback is invoked with a calculated value but the calculation is not defined. I would assume that you mean the list of available playback devices is non-empty :)
[1] https://w3c.github.io/remote-playback/#getting-the-remote-playback-devices-availability-information
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.