wicg / audio-focus Goto Github PK

I'm opening this issue as a follow up to an issue I opened in the Media Session API repo, asking about how that spec would play part in a PWA-replication of a native audio app like Spotify or a podcast app: w3c/mediasession#232

As I somewhat suspected I got pointed here.

Related, there is also an open issue about making the Web Audio API available to Workers: WebAudio/web-audio-api#1098

There's also an issue added to Project Fugu: https://bugs.chromium.org/p/chromium/issues/detail?id=997514

What I basically want to be working towards is being able to create a Progressive Web App podcast/audio player/radio app which can function as seamlessly as a native such player (and which doesn't need to be a Single Page Application).

This would entail having the Service Worker own the audio playback and it interacting with the audio focus, the Media Session API and the Web Audio API to ensure that when eg. one downloaded podcast has been played, the next will start playing + being able to skip etc + deal with audio focus like a user would expect.

To have such playback work from a background thread, I'm imagining that one would also have to be asking the user for some permission, so that not every site out there start playing sound in the background independently of the web page itself.

How would that relate to this specification and is this specification still worked upon to some degree?

From @foolip on October 1, 2015 12:12

First discussed in w3c/mediasession#111 (diff)

This is blocked on @domenic adding some advice to https://w3ctag.github.io/design-principles/ which I will then follow.

Copied from original issue: w3c/mediasession#112

Working group decision to adopt

See results of call for consensus in Media WG (January 2023)

Proposal

The Media Working Group has recently discussed the Audio Focus API again. The proposal is in scope of its charter (listed as a potential normative deliverable). The group refined the proposal, see new explainer in a separate repository.

The group is now ready to adopt the proposal. Note that the intent is not to migrate this repository but rather to migrate the new one and archive this one.

The explainer details goals, use cases, and the currently proposed shape for the API along with code examples. The underlying repository contains an initial list of issues under discussion.

Ongoing technical constraints

What technical constraints will be added to user agents implementing this feature? Better integration and audio-mixing capabilities of websites with native apps, so they can play on top of each other, or play exclusively.

Will this feature be supported in all environments (desktop, mobile, tablets, TV, eBooks, automotive, etc.)? Yes.

Link to implementation experience and demos

The explainer describes situations where this API is needed and includes code samples.

Data

The proposal is in the "close the gap with native" ballpark, fixing cases where developers complained to browser vendors about behavior, e.g. on mobile devices when an audio session gets interrupted by a call. No further quantitative data gathered.

Security and Privacy

To be worked out. No major issue anticipated.

Accessibility

To be worked out. No major issue anticipated.

Internationalization

The proposed API should not have any implication.

From @mounirlamouri on June 29, 2015 15:44

This is mostly to keep track of the idea. I don't think we should work on that soon because it can be added later.

It might be easier for developers if the session were to expose the media session participants because the UA must know them. The current API would require web pages to keep track of these in parallel.

Copied from original issue: w3c/mediasession#80

From @annevk on July 1, 2015 16:50

A site would want to disable its own set of controls when "transient-solo" happens, but if the user did a hardware pause, the site would simply want to change the pause into a play button or some such.

Copied from original issue: w3c/mediasession#89

From @richtr on May 20, 2015 13:32

From the spec:

When playing media on the web, developers are currently forced to adopt a single default platform modality for playing all media content. On the other hand, native applications can access much richer media integration options with an underlying platform. On mobile devices, native application developers can request many different forms of media integration with the platform to obtain access to headphone buttons, lock screens and notification areas as needed. On desktop devices, native applications have access to keyboard media key events. Native application developers can specify the conditions in which media content should pause or duck on audio interruptions (i.e. pause or lower the volume for the duration of an interruption), continue playing out when application focus is lost or the device screen is switched off and interface with internal and external remote controllers.

If media sessions allow web developers to opt-in to custom platform-level media behavior on different platforms why do we insist on enforcing strict, arbitrary platform-level integration in the case that web media content has not opted-in to that?

Currently on mobile devices, by default, <audio> will continue playing out when the web browser is backgrounded and/or the device's screen is switched off. It may provide notification area controls and allow users to play and pause the audio content from the notification area. Clicking on the notification may bring the user back to the browser and, ideally, bring the tab making the noise to the foreground. It may display audio metadata on the homescreen, obtained from either the <audio> element or document metadata (such as document title and favicon). It may only allow only one media element to play out at a time or mix multiple media elements to play out at the same time. It may automatically pause <video> when the browser is backgrounded. ...or it may not do any or some of these things depending on which browser you try.

All of this behavior is a.) inconsistently provided across different web browsers, b.) completely magic in that it cannot be observed or controlled by web applications and c.) must be opted-out of (instead of having to opt-in to it in the first place) by web developers through the introduction of media sessions.

In line with the principles of the extensible web manifesto we must try to explain or remove this auto-magic behavior for default media handling by specifying how 'default' media playback should perform consistently across different web browsers and devices.

So what should we do? Describe the current magic of default media handling somehow? Choose a single sensible, consistent modality for default media handling on mobile devices (e.g. by default let's choose to treat all media as 'ambient' content)? Or should we just leave the magic of default media handling alone and not try to explain it in programmatic terms and continue to leave this up to implementors to decide what platform-level inter-operation and integration to provide by default for media content?

Copied from original issue: w3c/mediasession#41

From @alastor0325 on January 6, 2016 10:3

https://mediasession.spec.whatwg.org/#media-session-kinds

Displays lock-screen and notification area user interfaces when playback begins.

Hi, all,
I would like to discuss the behavior of the "user interfaces".
Here are some questions, thanks :)

• This interface is used to notify user something is playing (like notification) or control the audio playback?
  • IMMO, I hope we can use it to control the audio. ex. play, pause, next song e.t.c.
• If this interface can be used for controlling audio, when should it appear/disappear?
  • appear : when the audio started (it should be no doubt)
  • disappear : (1) when the audio pause (2) when the audio end

Copied from original issue: w3c/mediasession#120

From @foolip on October 26, 2015 9:19

A MediaSession interacts with global state on the system, and if you have an active session that is interrupted, you make some guesses about what's going on. A short transient interruption is likely some kind of notification, a long transient interruption is probably a phone call.

We should also review whether MediaSession should be restricted to secure contexts.

(Inspired by @cbentzel on blink-dev, we can't ship without thinking this through.)

Copied from original issue: w3c/mediasession#114

From @mounirlamouri on June 29, 2015 15:55

AFAIK, WebRTC output is usually piped to a <video> that means that the element will be part of a MediaSession of some sort. However, even if it's close to content, it's not really content because it wouldn't make sense to pause the video when there is a notification or have a pause button in the device's lock screen.

We should try to see how we can make that UC fit into the spec model.

Copied from original issue: w3c/mediasession#81

From @jakearchibald on July 1, 2015 9:30

Do we do anything special with a <video> without an audio track?

Use case:

As a developer of an user-submitted video site, I want the videos to capture media focus if they have an audio track, but not if they don't.

Copied from original issue: w3c/mediasession#87

From @mounirlamouri on May 18, 2016 17:2

The API is designed in a way that we must get the kind right in order to implement other parts of the API. The reason is that kind is a parameter of MediaSession constructor and defaults to content so at least content type needs to be implemented in order to be forward compatible.

I wonder if we could introduce a default kind which would associate the created session with the default UA session. That way, a UA could not expose kind if it is not fully and properly implemented and expose other parts of the API. This behaviour would be feature detectable and forward compatible.

Practically, the rationale here is that MediaMetadata sounds like a great feature to ship on mobile but it is blocked by kind. The downside is that we are exposing an undefined behaviour in the spec and we will no longer make sessions defaulting to content which means developers will have to make a conscious decision if they want another type than default. Arguably, it can be a good thing.

@foolip, WDYT?

Copied from original issue: w3c/mediasession#128

I think this is a promising idea and I'd love to see it progress. Is this still being worked on?

From @alastor0325 on December 31, 2015 3:55

https://mediasession.spec.whatwg.org/#media-session-kinds

In-game sound effects and background music.

Hi, all,
I think the in-game sound is not very suit for the "ambient" type.

As "does not interact with any other "content", "transient", "transient-solo" or "ambient" content during playback".

Think about that situation, if user opens the web-game during listening content type music, then both of them would be playback. IMMO, It's not very good user experience.

Maybe can we have some new type can interact with other kinds but would be paused when its top-level browsing context is not visible?

Thanks.

Copied from original issue: w3c/mediasession#119

From @foolip on September 8, 2015 9:26

After w3c/mediasession#104 this table is non-normative, and so the description of the behaviors do not need to be so exhaustive. The normative requirements should be part of the various algorithms, and so a high-level summary of the behavior is sufficient in this context.

Copied from original issue: w3c/mediasession#105

From @foolip on March 8, 2016 16:8

https://w3c.github.io/remote-playback/

This is an API which can cause an individual media element to play remotely. Local and remote playback are not mutually exclusive, as it could make sense to play one video locally and another remotely at the same time, either with different people watching each, or more speculatively, with the two being different parts of the same experience, playing in sync.

Key issues:

• If a single media element is part of an active media session and begins playing remotely, what should happen?
• If multiple media elements are in an active media session and one of them begins playing remotely, what should happen?
• How the metadata used for local playback be reused for remote playback? Metadata is currently added at the session level but remote playback happens at the element level.

Copied from original issue: w3c/mediasession#123

From @annevk on June 15, 2015 11:18

https://mediasession.spec.whatwg.org/#indefinitely-pause

must invoke its pause() method

We should not invoke public methods from prose as they can be overridden by script. Please instead invoke the algorithm pause() invokes directly. This might involve filing a bug on the relevant specification to give that algorithm a name.

Copied from original issue: w3c/mediasession#65

Hi @mounirlamouri and @rebeccahughes, was there a discourse thread for this incubation? I can't seem to find it.

This repo seems to be missing some of the IPR templates, etc... which suggests it didn't go through the right channels to end up here. That's ok! but we should sort that out.

Also, was this not in the purview of the Media WG?

cc @WICG/chairs.

From @foolip on September 2, 2015 14:44

The spec currently defines some things in terms of what kind of media session caused a callback, with a fallback for when it was caused by something external. For example:

Let interrupting media session category be the media session category that triggered this interruption. If this interruption has no known media session category, let interrupting media session category be Default.

The way we want to implement media sessions, at least on Android, is instead to let each media session individually interact with the platform API, which implies that you would never know if an interruption comes from another media session or something else.

My proposed fix for this is to define a virtual platform API with roughly the following shape:

• A method to request audio focus of different kinds, and one to abandon it.
• Callbacks when you gain and lose audio focus, and the reason it happened.

Then define how an individual media session interacts with this system, instead of defining the interactions in terms of other media sessions.

Copied from original issue: w3c/mediasession#100

From @mounirlamouri on June 29, 2015 15:33

Ideally, we should allow MediaSession instances decide how to behave when they are resumed or interrupted. Exposing events that could be cancelled would help. This is particularly important for Web Audio for which the best a UA can do is to break the link between the producing source and the output device. A MediaSession instance might stop the producing source and resume it when needed.

In addition, the interrupt event could have information about the ducking behaviour.

Both events could have information regarding how and why the session was resumed/interrupted. For example, whether it happened from a user interaction.

Copied from original issue: w3c/mediasession#78