Decoding of timecode in canvas about opendatacam

[tdurand]

Yes, very true 😉 . Need to test that, could be a potential great perf improvement.

[tdurand]

Some first update on that ! Not 100% there yet.

1. Performance

Kudos @b-g ! this is a big win as expected.

Decoding timecode on each frame:

Decoding timecode on first frame + applying offset + rendering detections things on another canvas :

We went from a 18-20 FPS to a solid 40-45 FPS on my phone, an more important, we still have much room for other computations. And it opens also compatibility to worse phone than mine. (I have a samsung galaxy a3 2017, it's a mid-end perf), I have another shittier android phone , will try also.

On desktop (macbook pro 2012) we already had 60 FPS but now the CPU is just not doing anything.

2. Still to figure out

We never win that easy 🙃 , turns out that the video and the detection don't keep in sync.

You can test by opening the http://localhost:8080/x-playground/decode-once/ , (start http server on the repo via http-server command: https://www.npmjs.com/package/http-server ) .

Turns out that the offset between currentTiming and the timecodes of the detections is not constant. Seems that the playbackRate at the time of generation of the detections and the playbackRate of the video in the browser isn't the same. (the real offset is growing over time, this means that the video during detections had a faster playbackRate ? )

I'm now exploring to adjust the playbackRate to the timecodes, but maybe there is some fix that we can do on the detections side ? what do you think @b-g @mmmmmmmmmmmmmmmm ?

[b-g]

@tdurand Many thanks for the investigation! Yay!

1. ( I think the screenshots show both the same)

2. I believe the offset problem is a fundamental one, if we don't use a real "live" stream. All the current "detection videos" have been calculated "offline" from a pre-recorded video file, but stamped with absolute time according the time it took to detect the frames ... Hence I would like to propose the two different kind of encoded time codes: A) absolute millis for real live streams = what we use at the moment, and B) simple frame numbering for everything which deals with pre-recorded videos e.g. video with 25fps, first frame in video: num = 1, 1sec: num = 25 ... so that we always can calculate from video.currentTime the detection data ... we might even then need no encoded timecode at all.

[tdurand]

1. Updated, you can enjoy the peacefull flamechart 😉

2. Not sure I follow you 100% , I'm a bit struggling now to get around FPS of the browser, the video FPS, the timecode FPS ... ! Maybe a quick chat can help get my head clear.

[b-g]

Sorry + Sure! But today is tight ... can we do tomorrow in the morning e.g 10°°?

[tdurand]

Yes no problem, tomorrow is fine, 10 A.M it is ! If you have a moving schedule, I'll be on moovel stuff the whole day tomorrow also so no worry just ping me when available

[tdurand]

Some more investigations on that, I managed to get the video and the detections synced.

seems it's not a problem of flat playbackRate, but rather to keep up with the detections timecode we need to add a linear correction of the time .... that varies over time. A factor of ~0.57x.

To get the right timecode of the detection the formula is:

timecodeNow = initialOffsetBetweenVideoAndYolo + v.currentTime + correctionFactor * (v.currentTime - initialVideoTime)

I compute the correction factor like that: https://github.com/moovel/lab-traffic-cam/blob/master/x-playground/decode-once-synccorrector/index.html#L154 (I do it each 1s but it seems to be constant over time, so we could do it just once: )

But I'm struggling to understand why this corrector ???? Maybe missing something obvious here

You have two example to play with in the x-playground.

Also there is a little drawback of not having the canvas draw in sync the frame + detection, it is that once we get the right detection frame to draw, it then takes ~1-4ms to actually draw, and can slightly wrong , but if we "pre-add" 3-4 ms to the offset, we are fine. (this is a bit me being a bit picky...)

Let's talk about it tomorrow 😉

[b-g]

Hi @tdurand,

Could you give these files a try?

https://www.dropbox.com/s/6jprdabhlqsrd75/C0082-47mm.mp4?dl=0
https://www.dropbox.com/s/s35q8310zss9ra7/C0082-47mm-absolute.txt?dl=0

Basically:

• No timecode encoding the pixels
• Video runs with 25FPS and the detection result have an absolute frame index
• Use video.currentTime to calculate the current detection result

[tdurand]

Ok ! This works perfectly : http://localhost:8080/x-playground/no-decode-timecode/

And the code is super simple, no more canvas decoding 😄 !! It should have been our first prototype hehe, I like when we remove code !

But anyway, if we get down the road of live streaming, the time I spent writing the dynamic sync corrector will be useful 😆