Remote Playback API test automation about remote-playback

[louaybassbouss]

@Honry please let me know if can help on this

[Honry]

@mfoltzgoogle, @mounirlamouri, friendly ping. 😃

[anssiko]

ACTION: @louaybassbouss to figure out if we want to move forward with test automation or manual testing, to unblock PR publication.

via https://www.w3.org/2020/10/20-webscreens-minutes.html#a01

[FritzHeiden]

The new remote playback tests are now implemented. You can find the pull request here

The pull request is merged and tests are now available in master, however, there is still an open pull request for further style and text changes
web-platform-tests/wpt#41852

[Honry]

@louaybassbouss, I know you did really good on test automation for Presentation API, the problem we encountered in Remote Playback should be similar, could you please share your experience for us?

[Honry]

@louaybassbouss, friendly ping.

[louaybassbouss]

@Honry sorry I missed your request from 21 Jun thanks for the reminder. Yes the setup for testing the Presentation API and RemotePlayback API Tests on Chromecast should be similar. I would say that the tests of RemotePlayback API are easier to implement since you will not deal with different Presentation URLs as in the Presentation API (You can have a look in [1] where two different Presentation URLs are specified, one for cast receivers and one for other receivers that can render a web page). In RemotePlayback API you don't need to pass any URL which makes a lot of things easier. To test on Chromecast (or any other cast receiver like Android TV) you need just to make sure that your PC and the Receiver devices are in the same Network. But you need to keep in mind when you are writing the tests to not use any code that is specific for chromecast or any specific technology, but only according the API specification. To create a test report of RemotePlayback API e.g. for Chrome Browser, the tester (for example the person who is responsible to create the report) must have the information about which receiver devices are supported by the Browser under test. The tester musst also ensure that a receiver device is available during the test. I would recommend to habe a look to the tests of Presentation API in [2] and the current Test Report in [3].

PS: when you are writing tests please keep in mind that you are testing the API Implementation and NOT the underlying protocols.

[Honry]

@louaybassbouss, really appreciate for your exhaustive sharing, I agree with you that tests should be strictly compliance with API specification without any dependence. I will dig into tests of Presentation API as a reference to write tests for Remote Playback API.

[anssiko]

@Honry, we're likely to discuss this topic at our TPAC F2F on Monday 6 November, so any input, testing-related issues, even if preliminary, prior to that would be appreciated.

[Honry]

Can we talk about the possibility to implement Testing API(quite similar to w3c/presentation-api#440) to break through test automation?

[Honry]

Considering the Remote Playback API is much more simple then Presentation API, that is the amount of manual tests is much less. But implement a Testing API would bring much more efforts.

[mfoltzgoogle]

If the test API is strictly about testing the RemotePlayback interface, it could be very simple:

<video>.remote.test.deviceAvailable = true to simulate a remote playback device being available for that element, and false to simulate a device not being available.

If we want to test behavior of the media element itself in remote playback scenarios, that could mean creating a fake remote playback device with specific capabilities.

[Honry]

I think this should be enough. @louaybassbouss, what's your opinion?

[louaybassbouss]

@Honry <video>.remote.test.deviceAvailable = true is enough for Observing remote playback device availability but I am not sure how to test the RemotePlayback Connection State. We may need a 'FakePlaybackScreen' similar to this proposal from @mfoltzgoogle. The 'FakePlaybackScreen' can have some additional attributes like playing, paused, etc. to test the transition between the different remote playback connection states. What do you think?

[Honry]

@louaybassbouss, I was thinking using <video>.remote.test.deviceAvailable = true/false to monitor state change should be enough. But yes, I agree with you that we should also test the video state must be align with the remote playback.

[mfoltzgoogle]

We do need to be able to test a failure to connect to a remote playback device (step 4 of "Establishing a Connection.")

As far as testing the behavior of the media element, the mandatory requirements in the spec are:

• The user agent MUST send media commands to the remote playback device - but how would this be observable from a test?
• The media element MUST reflect changes to the remote playback state, which is similar to testing media playback in general.

Let's discuss this further at the F2F.

[mounirlamouri]

Blink is using a method to force update the device availability. Because we run this only in our tests which do not have a Remote Playback backend, I think the WPT method should be different as there will be an idea of forcing the value on top of the backend. Because of this, I would recommend the method to take true, false and null in order to unset the override. A method might make things clearer to (such as setOverrideDeviceAvailability()). Same could apply for connection state.

[mounirlamouri]

@Honry what's the status of the tests for Remote Playback API? It's my understanding that we are only testing non-user gesture and non-device specific parts of the API which isn't a very large amount. Are there plans to have a WPT API to mock a device?

[Honry]

Oops, I missed this issue. @mounirlamouri, thank you for pinging.

Defining a WebDriver extension API is a more standardized way and which has already been adopted in Generic Sensor, Permissions API, Reporting API, etc.. Moreover, Presentation API also advocated it at
w3c/presentation-api#440 (comment).

@anssiko, considering Remote Playback API is CR now, is that possible to roll back it with new WebDriver extension API?

[anssiko]

We can publish a revised CR with WebDriver extension API similarly to Generic Sensor API.

@tidoust for comments.

[tidoust]

@tidoust for comments.

I confirm that the group can publish a revised CR with a WebDriver extension API.
In any case, the implementation report needs to cover more than "non-user gesture and non-device specific parts of the API".

[Honry]

Thanks @tidoust, @anssiko for confirming this.

Since I will be OOO next two weeks for Chinese New Year holiday, I will figure out an initial proposal for review after the holiday.

[Honry]

@mounirlamouri, @anssiko, @mfoltzgoogle, @louaybassbouss,

Here is a draft WebDriver based API which I am thinking should be satisfied checkpoints you mentioned before, but I may have missed something, PTAL, all comments are very welcome!

https://github.com/Honry/remote-playback-test-automation/blob/master/webdriver-extension-api.md

One open: how many playback states should be covered?

(Sorry for late due to nCov situation)

[anssiko]

Thanks @Honry! (And no problem for the delay.)

@mfoltzgoogle @mounirlamouri can either of you help review the proposed WebDriver Extension API and give advise on the playback states question?

[mounirlamouri]

Thank you @Honry for the proposal. At a first glance it looks fine but I think we will have to implement and use that in Chrome to give more useful feedback. Do you intend to implement the test api in a browser?

[Honry]

Agree, but I'm not an expert in feature implementation, maybe @anssiko could help looking for someone interested in this. :)

BTW, @mounirlamouri, could you give advise on following open? I am thinking of picking several typical playback states should be sufficient.

One open: how many playback states should be covered?

[anssiko]

I think @foolip knows folks who touch Web Driver bits in Chromium. I don’t think we have someone readily available for this work at the moment. I suggest open a crbug and link it here.

[foolip]

If you'd like to implement https://github.com/Honry/remote-playback-test-automation/blob/master/webdriver-extension-api.md in ChromeDriver, then @JohnChen0 should be able to review.

For help testing things in WPT more generally, I would ping @stephenmcgruer @LukeZielinski @Hexcles.

[louaybassbouss]

Alternative proposal to move forward with Remote Playback Testing using a manual approach. This proposal is not replacing the WebDriver approach discussed in this thread, but just an alternative to Fake Remote Playback device.

• We use prepared Mezzanine Content to play on Remote Playback Device. The Mezzanine Content shows QR Code on each video frame which contains meta data about the timing/... . This CTA WAVE project can be used for this purpose.
• The Remote Playback Tests are implemented for control device in a way similar to the Presentation API tests. While a Test is running, the screen of the remote playback device will be recorded. This CTA WAVE Project includes tests following this approach.
• Test results are generated after the recording is analysed. This CTA WAVE project can be used for this purpose.

Lets discuss this option in our F2F meeting in few minutes.

[FritzHeiden]

The new remote playback tests are now implemented. You can find the pull request here