depth_from_video_in_the_wild: image size for pretrained models about google-research

[gariel-google]

@studennis911988 Sorry it's taking a bit long. This week is a holiday week, and things take time to get approved, so I am aiming for getting the code out in two weeks. Sorry again for the slowness.

…

On Fri, Nov 22, 2019 at 3:17 PM Dennis Cychuang ***@***.***> wrote: Yes, it does. It helps a lot. We can release the code for training on EuRoC, which would read from time-sorted sequences of files, will use no masks and will learn the camera matrix. Please confirm that this is what you'd need. It may take a week or two to release. … <#m_8087750674146224854_> @gariel-google <https://github.com/gariel-google> Hi, thanks for your open-source code and can I know is there any schedule on releasing the code for training EuRoc dataset? Thanks in advanced. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNHHJ6PUMMHKPZV4573QVBR7FA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE7EPAA#issuecomment-557729664>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNAOZUGKFBFGJ7XLCW3QVBR7FANCNFSM4I73I7CA> .

[gariel-google]

Here is the link to the paper http://openaccess.thecvf.com/content_ICCV_2019/papers/Gordon_Depth_From_Videos_in_the_Wild_Unsupervised_Monocular_Depth_Learning_ICCV_2019_paper.pdf And here is the supplementary material http://openaccess.thecvf.com/content_ICCV_2019/supplemental/Gordon_Depth_From_Videos_ICCV_2019_supplemental.pdf

…

On Mon, Nov 4, 2019 at 10:36 PM Quei-An Chen ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> Could you release the full paper that you submitted to ICCV? I downloaded the paper from ICCV QR code but can't find the supplementary matrials. Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNGMKW4YZC47AM3UZE3QSEH6DA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDBX6ZI#issuecomment-549683045>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNGODL4BONFEUSTAYLLQSEH6DANCNFSM4I73I7CA> .

[gariel-google]

Absolutely, we will release the model and the code. SO sorry it is taking long, I am swamped with work, but I am committed to this. Sorry again for the delay.

…

On Tue, Dec 10, 2019 at 8:09 PM Beniko_J ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> Hi, thanks for your work! I am also looking forward to seeing your training code for the EuRoC dataset. BTW, is it possible for you to release the trained model for the EuRoC dataset too? Best regards — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNFZKDOYOIHNITXFXMTQYBRZLA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGRZ76Y#issuecomment-564371451>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNHQZKNLBTYXKEFPLCTQYBRZLANCNFSM4I73I7CA> .

[gariel-google]

Hi @kwea123, Yes, it was trained at 416x128. The main reason that we stayed with 416x128 is that we wanted to not change too many things at the same time, so that we can study the effect of each change we made - occlusion-awareness, learning intrinsics, pooling datasets etc. For getting the best depth prediction accuracy, I would definitely train at higher resolutions. We could easily double the resolution (and have a batch size of 4 instead of 16) and everything would still fit on a p100 or a v100 GPU and train.

…

On Fri, Oct 11, 2019 at 8:31 AM kwea123 ***@***.***> wrote: Hi @gariel-google <https://github.com/gariel-google>, are the models that you provide trained on images 416x128? When I tried inference with other resolutions it doesn't work well at all. If it's indeed 416x128, have you tried training with higher resolutions? I know some previous work use 416x128 for training, but recently most methods use higher resolutions and experiments have demonstrated higher resolutions lead to better results. Is it something related to the GPU memory issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNCD5ZISHGFVPPUDRMTQOCL5HA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HRHXKGA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNDYD7HIU3ZPJQVYHG3QOCL5HANCNFSM4I73I7CA> .

[cognitiveRobot]

Hi @gariel-google, are the models that you provide trained on images 416x128? When I tried inference with other resolutions it doesn't work well at all.

If it's indeed 416x128, have you tried training with higher resolutions? I know some previous work use 416x128 for training, but recently most methods use higher resolutions and experiments have demonstrated higher resolutions lead to better results. Is it something related to the GPU memory issue?

Hi, did you retrain for higher resolution? If so, can you share the training script? Thanks.

[kwea123]

no, I just tested the inference.

[kwea123]

@gariel-google Could you release the full paper that you submitted to ICCV? I downloaded the paper from ICCV QR code but can't find the supplementary matrials. Thank you.

[cognitiveRobot]

@gariel-google Can you release the code to train the model from image sequence only. Thanks in advance.

[gariel-google]

Hi Md Z Hossain, Thanks for reaching out. I'm not sure I understand your question - image sequences only as opposed to what? You mean without segmentation masks? Our code otherwise is image sequences only.

…

On Tue, Nov 5, 2019, 12:09 PM Md Z Hossain ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> Can you release the code to train the model from image sequence only. Thanks in advance. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNB637C6J7W4FYRNYODQSHHGRA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDEE2GY#issuecomment-549997851>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNF4D6LIH7VIWDHUAPTQSHHGRANCNFSM4I73I7CA> .

[cognitiveRobot]

@gariel-google thanks for your quick reply. Looks like this current training code will only work if we provide the following three items.

1. a png file which is a stitched image from three consecutive frames.
2. a camera matrix file.
3. a png file is a mask image for moving objects.

But, I have only image frames.

In the paper, I find
@ 5.1 EuRoC Micro Aerial Vehicle Dataset:
.............. Vicon scene 3d scans, and camera calibration, we only used the monocular videos for
training....
So, I thought you have some other training code for only image frames. Does it make sense?

[gariel-google]

Yes, it does. It helps a lot. We can release the code for training on EuRoC, which would read from time-sorted sequences of files, will use no masks and will learn the camera matrix. Please confirm that this is what you'd need. It may take a week or two to release.

…

On Tue, Nov 5, 2019 at 1:53 PM Md Z Hossain ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> thanks for your quick reply. Looks like this current training code will only work if we provide the following three items. 1. a png file which is a stitched image from three consecutive frames. 2. a camera matrix file. 3. a png file is a mask image for moving objects. But, I have only image frames. In the paper, I find @ 5.1 EuRoC Micro Aerial Vehicle Dataset: .............. Vicon scene 3d scans, and camera calibration, we only used the monocular videos for training.... So, I thought you have some other training code for only images frames. Does it make sense? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNAV4A5JCNZBNAK2BVTQSHTNNA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDEO2SY#issuecomment-550038859>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNDTQAX2QPHDOMTCSKTQSHTNNANCNFSM4I73I7CA> .

[cognitiveRobot]

Yes. That's the code. :) I would be grateful if you release them. Thanks.
By this time, I made segmentation masks (easy for me, as there is no moving objects in my case) and fake camera matrix (by guessing the focal length and the principal point).

Unfortunately, it's not working. The error message I get

I1106 11:13:31.460314 140258611402496 train.py:164] Attempting to resume training from depth_from_video_in_the_wild/kitti_learned_intrinsics/model-248900...
I1106 11:13:31.460670 140258611402496 train.py:166] Last checkpoint found: None
I1106 11:13:31.460769 140258611402496 train.py:173] Training...
INFO:tensorflow:Error reported to Coordinator: <class 'AttributeError'>, 'dict' object has no attribute 'iteritems'
I1106 11:14:45.539946 140258611402496 coordinator.py:224] Error reported to Coordinator: <class 'AttributeError'>, 'dict' object has no attribute 'iteritems'

[gariel-google]

Sorry about that, this is probably due to a python 2 / python 3 issue ( https://www.tutorialspoint.com/What-is-the-difference-between-dict-items-and-dict-iteritems-in-Python). I seems unrelated to our code, I guess you can use python 2 or use a more advanced version of tensorflow?

…

On Tue, Nov 5, 2019 at 2:21 PM Md Z Hossain ***@***.***> wrote: Yes. That's the code. :) I would be grateful if you release them. Thanks. By this time, I made segmentation masks (easy for me, as there is no moving objects in my case) and fake camera matrix (by guessing the focal length and the principal point). Unfortunately, it's not working. The error message I get I1106 11:13:31.460314 140258611402496 train.py:164] Attempting to resume training from depth_from_video_in_the_wild/kitti_learned_intrinsics/model-248900... I1106 11:13:31.460670 140258611402496 train.py:166] Last checkpoint found: None I1106 11:13:31.460769 140258611402496 train.py:173] Training... INFO:tensorflow:Error reported to Coordinator: <class 'AttributeError'>, 'dict' object has no attribute 'iteritems' I1106 11:14:45.539946 140258611402496 coordinator.py:224] Error reported to Coordinator: <class 'AttributeError'>, 'dict' object has no attribute 'iteritems' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNBBJTNWTKDREPFYXE3QSHWVRA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDERLUY#issuecomment-550049235>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNBKULTW3ILIVRRIWM3QSHWVRANCNFSM4I73I7CA> .

[cognitiveRobot]

Ok. I will check with python 2. Thanks.

[cognitiveRobot]

Yeah. It works with python2. Thanks again @gariel-google

[cognitiveRobot]

@gariel-google I going to train on my images, but not sure exactly how to make samples.
I have sequence, lets say, img1, img2, img3, img4, img5, img6...
sample1: img1, img2, img3
sample2: img2, img3, img4
sample3: img3, img4, img5
and so on

I appreciate your reply.
Thanks

[gariel-google]

Hi Md, It sounds like you got it right. You concatenate the triplets along the width dimension, as exemplified here <https://github.com/google-research/google-research/tree/master/depth_from_video_in_the_wild/data_example/erfurt_93> .

…

On Sun, Nov 10, 2019 at 2:19 PM Md Z Hossain ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> I going to train on my images, but not sure exactly how to make samples. I have sequence, lets say, img1, img2, img3, img4, img5, img6... sample1: img1, img2, img3 sample2: img2, img3, img4 sample3: img3, img4, img5 and so on I appreciate your reply. Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNFQ4PPEOMBKOXPLMRLQTCCFPA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDVIPDY#issuecomment-552241039>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNFEEFNA3UV4KMXJGO3QTCCFPANCNFSM4I73I7CA> .

[studennis911988]

Yes, it does. It helps a lot. We can release the code for training on EuRoC, which would read from time-sorted sequences of files, will use no masks and will learn the camera matrix. Please confirm that this is what you'd need. It may take a week or two to release.
…

@gariel-google Hi, thanks for your open-source code and can I know is there any schedule on releasing the code for training EuRoc dataset? Thanks in advanced.

[Beniko95J]

@gariel-google Hi, thanks for your work! I am also looking forward to seeing your training code for the EuRoC dataset. BTW, is it possible for you to release the trained model for the EuRoC dataset too?

Best regards

[studennis911988]

@gariel-google Really appreciate for your hard work, however I have a little problem here.
Since I wish to get the real depth data from the disparity map produced by the model, I'm wondering is it right to calculate the depth by the traditional stereo disparity to depth formula ? Or there is another way to do that.
Thanks in advanced!

[cognitiveRobot]

@gariel-google Hi, any update on releasing the code and the checkpoint file? Thanks

[studennis911988]

@gariel-google Thanks for all your hard work on releasing the code for Euroc dataset, but could you provide some instructions about it in readme section?
Thanks in advanced!

[gariel-google]

Sure, I will add the links and instructions over the next few days.

…

On Thu, Feb 6, 2020 at 5:28 PM Dennis Cychuang ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> Thanks for all your hard work on releasing the code for Euroc dataset, but could you provide some instructions about it in readme section? Thanks in advanced! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNCT3GC354L6OUHVHP3RBS2MTA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELBM3FY#issuecomment-583191959>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNC2AUUKP5AH2I5MJFLRBS2MTANCNFSM4I73I7CA> .

[gariel-google]

The checkpoints and links are out.

…

On Fri, Feb 7, 2020 at 9:53 AM Ariel Gordon ***@***.***> wrote: Sure, I will add the links and instructions over the next few days. On Thu, Feb 6, 2020 at 5:28 PM Dennis Cychuang ***@***.***> wrote: > @gariel-google <https://github.com/gariel-google> Thanks for all your > hard work on releasing the code for Euroc dataset, but could you provide > some instructions about it in readme section? > Thanks in advanced! > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#61?email_source=notifications&email_token=ADXKUNCT3GC354L6OUHVHP3RBS2MTA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELBM3FY#issuecomment-583191959>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ADXKUNC2AUUKP5AH2I5MJFLRBS2MTANCNFSM4I73I7CA> > . >

[studennis911988]

@gariel-google Thanks for the explanation, but I still have two questions about render the depth from EuRoc dataset.
First of all, how can I get the real depth value from the depth image produced by the render_euroc_depth.py?
Second, I had noticed that I don't have to provide model (checkpoints) to run the script(render_euroc_depth.py), though it still give the depth images, so I'm wondering if I use the wrong script ?
Thanks for your attention!

[gariel-google]

render_euroc_depth.py renders a depth map from the grountruth, not form a model / checkpoint. The groundtruth in EuRoC is in the form of a fused point cloud, a single point cloud for the entire room. We need to render a depth map out of that, given a certain camera position. That's what render_euroc_depth.py does. It projects the fused point cloud onto a given camera position, removes points that are out of frame and occluded points (of course, for the latter we make some assumptions about the angular width of a point depth discontinuity etc), and finally, resamples the points more or less evenly in image space. Since all these steps require some assumptions approximations, we were asked to provide a script that renders the groundtruth depths in the same way as we did in the paper. Does it make sense?

…

On Tue, Feb 11, 2020 at 11:53 PM Dennis Cychuang ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> Thanks for the explanation, but I still have two questions about render the depth from EuRoc dataset. First of all, how can I get the *real* depth value from the depth image produced by the render_euroc_depth.py? Second, I had noticed that I don't have to provide model (checkpoints) to run the script(render_euroc_depth.py), though it still give the depth images, so I'm wondering if I use the wrong script ? Thanks for your attention! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNBWHAWLJPEKGJ7GKM3RCOTHXA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELPZDXQ#issuecomment-585077214>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNFRQDDYSIG225PIGETRCOTHXANCNFSM4I73I7CA> .

[studennis911988]

@gariel-google Thanks for the detailed explanation!

If I understand correctly, the render_euroc_depth.py is just a script that provides groundtruth depth map for evaluating the learned depth map which is produced by trained model as mentioned in the paper.

As now I'm wondering how can I use the downloaded EuRoC cam0 data to produced the depth map from the model(checkpoint you provide for all 11 sequences) , moreover, is the depth map produced by model gives as the true real world depth value ?( since I have noticed that the groundtruth depth gives us some strange depth value like nagative 26 or positive 198)

Thanks again!

[gariel-google]

As far as I understand, it gives true depth, because the EuRoC point clouds are metric. Regarding the strange values: Negative values should be pruned here: https://github.com/google-research/google-research/blob/master/depth_from_video_in_the_wild/render_euroc_depth.py#L94, so I don't know where they are coming form. If you could help in debugging (by stepping into the script and seeing how come the negative depths survive the filter) that would be of great help!

…

On Wed, Feb 12, 2020 at 5:30 PM Dennis Cychuang ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> Thanks for the detailed explanation! If I understand correctly, the render_euroc_depth.py is just a script that provides groundtruth depth map for evaluating the learned depth map which is produced by trained model as mentioned in the paper. So now I'm wondering how *can I use the downloaded EuRoC cam0 data to produced the depth map from the model* , moreover, is the depth map produced by model gives as the *true real world depth value* ?( since I have noticed that the groundtruth depth gives us some strange depth value like nagative 26 or positive 198) Thanks again! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=ADXKUNDNDYW6EXAUAMHSXE3RCSPCTA5CNFSM4I73I7CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELTBK7Q#issuecomment-585504126>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNBYIALDHEL7XM5N4TDRCSPCTANCNFSM4I73I7CA> .

[studennis911988]

@gariel-google
Thanks for the clarification, and I'll look deep in to the code to find the bug.
By the way, which script should I run to use the checkpoints you provided for all 11 sequences to produce learned depth images?

[studennis911988]

@gariel-google Sorry for bothering you, I'm wondering is there father update for euroc dataset ?

[gariel-google]

@studennis911988 Sorry for the delayed response, I was sprinting for ECCV, then out of office. This <https://github.com/tensorflow/models/blob/master/research/struct2depth/inference.py> script form struct2depth should do the job. You'd have to change their model <https://github.com/tensorflow/models/blob/master/research/struct2depth/model.py> to ours <https://github.com/google-research/google-research/blob/master/depth_from_video_in_the_wild/model.py>, but otherwise everything should be compatible. Please let me know how it goes!

…

On Tue, Mar 17, 2020 at 5:43 PM Dennis Cychuang ***@***.***> wrote: @gariel-google <https://github.com/gariel-google> Sorry for bothering you, I'm wondering is there father update for euroc dataset ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXKUNDWJYJ5RNKUFCCGFZDRIAKLXANCNFSM4I73I7CA> .