Weird decoding glitch with AV1 about moonlight-qt

[makedir]

Seems someone already suggested it a year ago here with the same issue, that the stream lags on no motion with no answer:

LizardByte/Sunshine#1610

Is there no option for Sunshine to force constant FPS?

[makedir]

"Hold on a second... Does that huge decoding latency (50-70ms) really affects the input or audio in the game? For example, when you start scrolling or zooming in/out on the map do you experience some sort of severe glitching/shuttering?"

Yes?? of course. the entire stream laggs when this happens. or mouse cursor.

[mvillarg]

To me this looks like a bug on the Intel 12th gen hardware decoder when the image doesn't change a bit from frame to frame. Can you open Windows performance stats and take a look at both CPU % utilization and GPU decoder graph when this happens?

[makedir]

@mvillarg how? you mean perfmon? I dont see anything in there to frack gpu usage or even gpu video decode usage.

[makedir]

I was able to reproduce the issue with running HwInfo, being on desktop, and open a https://www.testufo.com/ to trigger a constant 120fps animation on screen and then switch between two tabs in Firefox between ufotest and no motion.

The moment I change to ufotest and have constant 120fps animation it seems to work fine, the moment I change to other tab and have no motion on screen, the problem starts to happen, but JUST if I also slowly move the mouse cursor around very slowly. When I dont move the mouse cursor around it doesnt seem to happen.

Pxl.20240531.101014545.mp4

Pxl.20240531.101014545b.mp4

This doesnt happen with h264 and h265, just AV1. Not sure if the issue is on the client side with Moonlight or on the server with Sunshine.

[mvillarg]

I mean you can open Windows's Task manager which has well-known performance stats for CPU, RAM, GPU and Network. When you detect the issue you can alt+tab and see what's happening there. The GPU stats include 3D workload performance, VRAM usage and both enconding & decoding performance charts.

[makedir]

@mvillarg please see what I posted on 2nd post. Issue also seems to happen with h265 but just very rarely and just for 1 second or so randomly and then goes back, also just when moving mouse around. What is weird though is, even with high mouse movement over entire screen, the issue wont dissolve with AV1 and is stuck at high latency.

Is there a way on host / Sunshine to force a constant 120FPS stream even with no motion? It seems the host is throttling FPS down to even 11fps with no motion, maybe that is the issue and the decoder gets bugged with that. Though it doesnt seem to bug out that much on h265 just av1.

[mvillarg]

Hi @makedir ,
Ok, this is simpler than it looks. What's happening here is not actually related to AV1 not being able to encode at a constant bitrate, but simply that the game is not sending draw commands to the GPU, hence Sunshine is not encoding/sending anything to Moonlight. This even occurs in Windows itself, where if you don't move the mouse the FPS counter almost drops to 0.

[makedir]

@mvillarg and you want now what say with that?

1. why doesnt it happen with h264 and h265
2. why is this also just happening if moving mouse cursor slowly like seen in the examples in combination with no other screen motion. what you said there is wrong. look at the examples again. it does not happen if not moving the mouse it just happens if you move mouse at the same time of no other draw. also moving mouse quickly doesnt resolve it, where it should with your theory.
3. why is there no option in sunshine forcing a constant fps / constant bitrate
4. what is the solution / workaround for this
5. so you are saying moonlight and sunshine are useless? because it is, if they behave like this with no constant fps force even with no draw on screen.

[mvillarg]

Hold on a second... Does that huge decoding latency (50-70ms) really affects the input or audio in the game? For example, when you start scrolling or zooming in/out on the map do you experience some sort of severe glitching/shuttering?

If you don't experience such a high input lag or audio shuttering in that specific situation where the game is not rendering any new frame I wouldn't worry too much. The high decoding time might be tightly related to both hardware or drivers issues on the N100, or just the way delta time is measured in thouse super low framerate conditions between Sunshine and Moonlight.