Recommended way of extracting background in AR mode about webxr-samples

[klausw]

@ArnaudHambenne is right, the current Chrome immersive-ar implementation does not allow JS-side access to camera pixels. For the future, my understanding is that there should be a separate API or feature that provides pose-aligned camera images, where applications would need to request such access at session start so that the user agent can show appropriate consent prompts.

See https://github.com/immersive-web/computer-vision which mentions this as a use case. Previous discussion was also in immersive-web/webxr#694 (comment) and immersive-web/proposals#36 .

(I also had some thoughts on this in a technical paper at https://www.tdcommons.org/dpubs_series/1902/ . That isn't intended to represent any particular future directions or plans, though I'm still partial to the dinosaur sketches.)

[bialpio]

There have been some efforts to make this happen. Very early prototype is available in Chrome, behind WebXRIncubations flag - the API is described here with a sample usage demonstrated here, but please note that the API is potentially unstable and in no way final.

[ArnaudHambenne]

Hi @ArnaudHambenne ,

.readPixels() doesn't seem to work as well. What browser did you use and what device?

My code is something like -

  function onXRFrame(t, frame) {
        let session = frame.session;
        let gl = session.renderState.baseLayer.context;
        var pixels = new Uint8Array(gl.drawingBufferWidth * gl.drawingBufferHeight * 4);
        gl.readPixels(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
        if (pixels.every( (val, i, arr) => val === arr[0] ) ){
          console.log("none");
        }else{
            console.log(pixels);
        }
  }

What did you use to read the pixels?

I am not too worried about the performance hit as I would only like to read the frames at specific intervals. Could you please let me know if you used any other flag to access the pixel values?

Hi. It's been a while since I've worked on this API, although last time I did they had just released a major update stripping all extraction capabilities of the WebXR buffers out of privacy concerns. I'm not sure if they've come around on that, but if they haven't this means that neither readPixels() nor DrawImage() nor TransferToImageBitmap() can be used to extract the pixels from the buffers. All these functions will return an array of zeroes, so viewing any content of the buffers is off the table. I did however read that there was a group of people advocating for some kind of computer vision capabilities, but not sure how far they've come on that. Your code is probably fine.

[pavan4]

Hi @ArnaudHambenne ,

.readPixels() doesn't seem to work as well. What browser did you use and what device?

My code is something like -

  function onXRFrame(t, frame) {
        let session = frame.session;
        let gl = session.renderState.baseLayer.context;
        var pixels = new Uint8Array(gl.drawingBufferWidth * gl.drawingBufferHeight * 4);
        gl.readPixels(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
        if (pixels.every( (val, i, arr) => val === arr[0] ) ){
          console.log("none");
        }else{
            console.log(pixels);
        }
  }

What did you use to read the pixels?

I am not too worried about the performance hit as I would only like to read the frames at specific intervals. Could you please let me know if you used any other flag to access the pixel values?

[pikilipita]

Are there any news on this issue?
It's a shame no to be able to let the user record a video or take a screenshot of his/her webXR session.