rpautrat / superpoint

Efficient neural feature detector and descriptor

SuperPoint

This is a Tensorflow implementation of "SuperPoint: Self-Supervised Interest Point Detection and Description." Daniel DeTone, Tomasz Malisiewicz, Andrew Rabinovich. ArXiv 2018.

Update: We have converted the Tensorflow model into a Pytorch one. This provides a version of SuperPoint with MIT license, which can be used with the Pytorch model defined in superpoint_pytorch.py.

Results on HPatches

Detector evaluation

Repeatability on HPatches computed with 300 points detected in common between pairs of images and with a NMS of 4:

	Illumination changes	Viewpoint changes
SuperPoint (our implementation)	0.662	0.674
SuperPoint (pretrained model of MagicLeap)	0.641	0.621
FAST	0.576	0.625
Harris	0.630	0.755
Shi	0.584	0.629

Descriptors evaluation

Homography estimation on HPatches computed with a maximum of 1000 points detected in common between pairs of images, a threshold of correctness of 3 and with a NMS of 8:

	Illumination changes	Viewpoint changes
SuperPoint (our implementation)	0.965	0.712
SuperPoint (pretrained model of MagicLeap)	0.923	0.742
SIFT	0.807	0.766
ORB	0.523	0.414

Homography estimation on HPatches computed with a maximum of 1000 points detected in common between pairs of images, with all kind of changes (viewpoint and illumination) and with a NMS of 8:

Correctness threshold	e = 1	e = 3	e = 5
SuperPoint (our implementation)	0.483	0.836	0.910
SuperPoint (pretrained model of MagicLeap)	0.438	0.833	0.914
SIFT	0.498	0.786	0.786
ORB	0.162	0.467	0.564

Installation

make install  # install the Python requirements and setup the paths

Python 3.6.1 is required. You will be asked to provide a path to an experiment directory (containing the training and prediction outputs, referred as $EXPER_DIR) and a dataset directory (referred as $DATA_DIR). Create them wherever you wish and make sure to provide their absolute paths.

MS-COCO 2014 and HPatches should be downloaded into $DATA_DIR. The Synthetic Shapes dataset will also be generated there. The folder structure should look like:

$DATA_DIR
|-- COCO
|   |-- train2014
|   |   |-- file1.jpg
|   |   `-- ...
|   `-- val2014
|       |-- file1.jpg
|       `-- ...
`-- HPatches
|   |-- i_ajuntament
|   `-- ...
`-- synthetic_shapes  # will be automatically created

Usage

All commands should be executed within the superpoint/ subfolder. When training a model or exporting its predictions, you will often have to change the relevant configuration file in superpoint/configs/. Both multi-GPU training and export are supported. Note that MagicPoint and SuperPoint only work on images with dimensions divisible by 8 and the user is responsible for resizing them to a valid dimension.

1) Training MagicPoint on Synthetic Shapes

python experiment.py train configs/magic-point_shapes.yaml magic-point_synth

where magic-point_synth is the experiment name, which may be changed to anything. The training can be interrupted at any time using Ctrl+C and the weights will be saved in $EXPER_DIR/magic-point_synth/. The Tensorboard summaries are also dumped there. When training for the first time, the Synthetic Shapes dataset will be generated.

2) Exporting detections on MS-COCO

python export_detections.py configs/magic-point_coco_export.yaml magic-point_synth --pred_only --batch_size=5 --export_name=magic-point_coco-export1

This will save the pseudo-ground truth interest point labels to $EXPER_DIR/outputs/magic-point_coco-export1/. You might enable or disable the Homographic Adaptation in the configuration file. Note that if you want to train your model with resized images, you have to export the detections on the resized images directly. You can resize the images with the parameter data->preprocessing->resize of the config file.

3) Training MagicPoint on MS-COCO

python experiment.py train configs/magic-point_coco_train.yaml magic-point_coco

You will need to indicate the paths to the interest point labels in magic-point_coco_train.yaml by setting the entry data/labels, for example to outputs/magic-point_coco-export1. You might repeat steps 2) and 3) several times.

4) Evaluating the repeatability on HPatches

python export_detections_repeatability.py configs/magic-point_repeatability.yaml magic-point_coco --export_name=magic-point_hpatches-repeatability-v

You will need to decide whether you want to evaluate for viewpoint or illumination by setting the entry data/alteration in the configuration file. The predictions of the image pairs will be saved in $EXPER_DIR/outputs/magic-point_hpatches-repeatability-v/. To proceed to the evaluation, head over to notebooks/detector_repeatability_hpatches.ipynb. You can also evaluate the repeatability of the classical detectors using the configuration file classical-detectors_repeatability.yaml.

5) Validation on MS-COCO

It is also possible to evaluate the repeatability on a validation split of COCO. You will first need to generate warped image pairs using generate_coco_patches.py.

6) Training of SuperPoint on MS-COCO

Once you have trained MagicPoint with several rounds of homographic adaptation (one or two should be enough), you can export again the detections on MS-COCO as in step 2) and use these detections to train SuperPoint by setting the entry data/labels:

python experiment.py train configs/superpoint_coco.yaml superpoint_coco

7) Evaluation of the descriptors with homography estimation on HPatches

python export_descriptors.py configs/superpoint_hpatches.yaml superpoint_coco --export_name=superpoint_hpatches-v

You will need to decide again whether you want to evaluate for viewpoint or illumination by setting the entry data/alteration in the configuration file. The predictions of the image pairs will be saved in $EXPER_PATH/outputs/superpoint_hpatches-v/. To proceed to the evaluation, head over to notebooks/descriptors_evaluation_on_hpatches.ipynb. You can also evaluate the repeatability of the classical detectors using the configuration file classical-descriptors.yaml.

Matching Features Demo with Pretrained Weights

A set of pretrained weights is provided for you labeled sp_v6. You will need to extract the weights and place the directory in your experiments directory so that the path $EXPER_PATH/saved_models/sp_v6 contains the weights. This can be easily done with the following command being run from the top level, SuperPoint, directory.

tar -xzvf pretrained_models/sp_v6.tgz $EXPER_PATH/saved_models/sp_v6

With the pretrained weights extracted, you can run the match_features_demo.py to compare SuperPoint and SIFT matches across two images:

python match_features_demo.py sp_v6 $DATA_PATH/HPatches/i_pool/1.ppm $DATA_PATH/i_pool/6.ppm

Note that you can also pass in additional arguments such as --H, --W, --k_best to specify the height and width to resize the images and the maximum number of keypoints you wish to keep from the detection process.

Fine-tuning a pretrained model

You can also directly reuse the weights of the pretrained model on MS COCO and fine-tune it with your own model. First download and unzip one of the pretrained models in your $EXPER_PATH folder:

• MagicPoint (synthetic)
• MagicPoint (COCO)
• SuperPoint (COCO)

Then launch a training using the pretrained model, for example to fine-tune SuperPoint:

python superpoint/experiment.py train superpoint/configs/superpoint_coco.yaml superpoint_finetuned --pretrained_model sp_v6

Credits

This implementation was developed by Rémi Pautrat and Paul-Edouard Sarlin. Please contact Rémi for any enquiry.

superpoint's Issues

The problem of Training of SuperPoint on MS-COCO

@rpautrat hello:
When i run the step6) "python3 experiment.py train configs/superpoint_coco.yaml superpoint_coco" using the branch, i got the following statements, Can you give me some advice? I really do not know how to solve this. Thank you...

[01/11/2019 00:29:25 INFO] Running command TRAIN
[01/11/2019 00:29:25 INFO] Number of GPUs detected: 1
[01/11/2019 00:29:28 INFO] Caching data, fist access will take some time.
[01/11/2019 00:29:29 INFO] Caching data, fist access will take some time.
[01/11/2019 00:29:30 INFO] Caching data, fist access will take some time.
2019-01-11 00:29:31.373688: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-01-11 00:29:31.441013: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-01-11 00:29:31.441328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.695
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 7.11GiB
2019-01-11 00:29:31.441340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2019-01-11 00:29:31.610840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6868 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
[01/11/2019 00:29:31 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:31 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
2019-01-11 00:29:33.541473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2019-01-11 00:29:33.541570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 135 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
[01/11/2019 00:29:41 INFO] Start training
2019-01-11 00:29:45.965791: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x7f8dc016dbb0
2019-01-11 00:29:57.274840: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.12GiB. Current allocation summary follows.
2019-01-11 00:29:57.274945: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): Total Chunks: 50, Chunks in use: 46. 12.5KiB allocated for chunks. 11.5KiB in use in bin. 5.1KiB client-requested in use in bin.
2019-01-11 00:29:57.274977: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): Total Chunks: 26, Chunks in use: 20. 13.5KiB allocated for chunks. 10.2KiB in use in bin. 10.0KiB client-requested in use in bin.
2019-01-11 00:29:57.275005: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): Total Chunks: 17, Chunks in use: 12. 17.8KiB allocated for chunks. 12.5KiB in use in bin. 12.0KiB client-requested in use in bin.
2019-01-11 00:29:57.275037: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275060: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275093: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): Total Chunks: 1, Chunks in use: 0. 15.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275118: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): Total Chunks: 1, Chunks in use: 0. 30.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275137: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): Total Chunks: 1, Chunks in use: 0. 56.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275157: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): Total Chunks: 1, Chunks in use: 0. 69.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275180: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): Total Chunks: 3, Chunks in use: 3. 432.0KiB allocated for chunks. 432.0KiB in use in bin. 432.0KiB client-requested in use in bin.
2019-01-11 00:29:57.275201: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): Total Chunks: 3, Chunks in use: 2. 828.0KiB allocated for chunks. 544.0KiB in use in bin. 544.0KiB client-requested in use in bin.
2019-01-11 00:29:57.275221: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): Total Chunks: 7, Chunks in use: 5. 4.90MiB allocated for chunks. 3.45MiB in use in bin. 3.45MiB client-requested in use in bin.
2019-01-11 00:29:57.275245: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): Total Chunks: 13, Chunks in use: 10. 20.95MiB allocated for chunks. 16.95MiB in use in bin. 16.95MiB client-requested in use in bin.
2019-01-11 00:29:57.275267: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): Total Chunks: 15, Chunks in use: 13. 49.42MiB allocated for chunks. 42.39MiB in use in bin. 40.43MiB client-requested in use in bin.
2019-01-11 00:29:57.275289: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): Total Chunks: 8, Chunks in use: 8. 57.09MiB allocated for chunks. 57.09MiB in use in bin. 56.25MiB client-requested in use in bin.
2019-01-11 00:29:57.275312: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): Total Chunks: 11, Chunks in use: 10. 149.70MiB allocated for chunks. 140.91MiB in use in bin. 140.62MiB client-requested in use in bin.
2019-01-11 00:29:57.275331: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275354: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): Total Chunks: 8, Chunks in use: 8. 455.36MiB allocated for chunks. 455.36MiB in use in bin. 450.00MiB client-requested in use in bin.
2019-01-11 00:29:57.275372: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275389: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275409: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): Total Chunks: 2, Chunks in use: 1. 5.99GiB allocated for chunks. 4.19GiB in use in bin. 4.12GiB client-requested in use in bin.
2019-01-11 00:29:57.275429: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 4.12GiB was 256.00MiB, Chunk State:
2019-01-11 00:29:57.275459: I tensorflow/core/common_runtime/bfc_allocator.cc:652] Size: 1.80GiB | Requested Size: 17.16MiB | in_use: 0, prev: Size: 4.19GiB | Requested Size: 4.12GiB | in_use: 1
2019-01-11 00:29:57.275481: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200000 of size 1280
2019-01-11 00:29:57.275498: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200500 of size 1280
2019-01-11 00:29:57.275522: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200a00 of size 256
2019-01-11 00:29:57.275545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200b00 of size 256
2019-01-11 00:29:57.275568: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200c00 of size 256
2019-01-11 00:29:57.275592: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200d00 of size 256
2019-01-11 00:29:57.275615: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200e00 of size 256
2019-01-11 00:29:57.275638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200f00 of size 256
2019-01-11 00:29:57.275660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201000 of size 256
2019-01-11 00:29:57.275676: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201100 of size 256
2019-01-11 00:29:57.275690: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201200 of size 256
2019-01-11 00:29:57.275710: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201300 of size 256
2019-01-11 00:29:57.275730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201400 of size 256
2019-01-11 00:29:57.275745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201500 of size 256
2019-01-11 00:29:57.275760: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201600 of size 256
2019-01-11 00:29:57.275777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201700 of size 256
2019-01-11 00:29:57.275812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201800 of size 256
2019-01-11 00:29:57.275831: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201900 of size 256
2019-01-11 00:29:57.275846: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201a00 of size 256
2019-01-11 00:29:57.275866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201b00 of size 256
2019-01-11 00:29:57.275889: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201c00 of size 256
2019-01-11 00:29:57.275914: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201d00 of size 256
2019-01-11 00:29:57.275939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201e00 of size 256
2019-01-11 00:29:57.275963: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201f00 of size 256
2019-01-11 00:29:57.275995: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202000 of size 256
2019-01-11 00:29:57.276012: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202100 of size 256
2019-01-11 00:29:57.276035: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202200 of size 256
2019-01-11 00:29:57.276059: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202300 of size 256
2019-01-11 00:29:57.276088: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202400 of size 589824
2019-01-11 00:29:57.276106: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a292400 of size 512
2019-01-11 00:29:57.276121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a292600 of size 512
2019-01-11 00:29:57.276136: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a292800 of size 589824
2019-01-11 00:29:57.276153: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a322800 of size 512
2019-01-11 00:29:57.276174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a322a00 of size 512
2019-01-11 00:29:57.276215: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a322c00 of size 512
2019-01-11 00:29:57.276232: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a322e00 of size 512
2019-01-11 00:29:57.276247: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a323000 of size 589824
2019-01-11 00:29:57.276262: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3b3000 of size 512
2019-01-11 00:29:57.276277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3b3200 of size 512
2019-01-11 00:29:57.276292: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a3b3400 of size 768
2019-01-11 00:29:57.276307: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3b3700 of size 512
2019-01-11 00:29:57.276322: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3b3900 of size 294912
2019-01-11 00:29:57.276337: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a3fb900 of size 256
2019-01-11 00:29:57.276352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3fba00 of size 512
2019-01-11 00:29:57.276367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a3fbc00 of size 512
2019-01-11 00:29:57.276382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3fbe00 of size 262144
2019-01-11 00:29:57.276398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43be00 of size 1024
2019-01-11 00:29:57.276421: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a43c200 of size 1024
2019-01-11 00:29:57.276442: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43c600 of size 256
2019-01-11 00:29:57.276467: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43c700 of size 256
2019-01-11 00:29:57.276499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43c800 of size 256
2019-01-11 00:29:57.276515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43c900 of size 256
2019-01-11 00:29:57.276537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43ca00 of size 256
2019-01-11 00:29:57.276561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43cb00 of size 512
2019-01-11 00:29:57.276587: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43cd00 of size 768
2019-01-11 00:29:57.276612: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a43d000 of size 1024
2019-01-11 00:29:57.276638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43d400 of size 256
2019-01-11 00:29:57.276658: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a43d500 of size 256
2019-01-11 00:29:57.276684: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43d600 of size 147456
2019-01-11 00:29:57.276711: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a461600 of size 256
2019-01-11 00:29:57.276731: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a461700 of size 1179648
2019-01-11 00:29:57.276756: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a581700 of size 1280
2019-01-11 00:29:57.276785: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a581c00 of size 1024
2019-01-11 00:29:57.276807: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a582000 of size 1024
2019-01-11 00:29:57.276835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a582400 of size 256
2019-01-11 00:29:57.276854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a582500 of size 1179904
2019-01-11 00:29:57.276879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6a2600 of size 256
2019-01-11 00:29:57.276906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a6a2700 of size 71168
2019-01-11 00:29:57.276927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6b3d00 of size 147456
2019-01-11 00:29:57.276954: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a6d7d00 of size 256
2019-01-11 00:29:57.276980: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6d7e00 of size 256
2019-01-11 00:29:57.277006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6d7f00 of size 256
2019-01-11 00:29:57.277026: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6d8000 of size 147456
2019-01-11 00:29:57.277057: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6fc000 of size 921600
2019-01-11 00:29:57.277078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a7dd000 of size 921856
2019-01-11 00:29:57.277094: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a8be100 of size 58112
2019-01-11 00:29:57.277120: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a8cc400 of size 62613248
2019-01-11 00:29:57.277140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020e482b00 of size 58982400
2019-01-11 00:29:57.277165: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10211cc2b00 of size 15360
2019-01-11 00:29:57.277191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6700 of size 256
2019-01-11 00:29:57.277208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6800 of size 256
2019-01-11 00:29:57.277223: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6900 of size 256
2019-01-11 00:29:57.277243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6a00 of size 256
2019-01-11 00:29:57.277258: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6b00 of size 256
2019-01-11 00:29:57.277273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6c00 of size 256
2019-01-11 00:29:57.277298: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6d00 of size 256
2019-01-11 00:29:57.277324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10211cc6e00 of size 256
2019-01-11 00:29:57.277343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6f00 of size 256
2019-01-11 00:29:57.277372: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7000 of size 256
2019-01-11 00:29:57.277400: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7100 of size 512
2019-01-11 00:29:57.277418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7300 of size 512
2019-01-11 00:29:57.277436: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7500 of size 512
2019-01-11 00:29:57.277467: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7700 of size 512
2019-01-11 00:29:57.277494: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7900 of size 512
2019-01-11 00:29:57.277515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7b00 of size 1024
2019-01-11 00:29:57.277542: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7f00 of size 512
2019-01-11 00:29:57.277567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8100 of size 512
2019-01-11 00:29:57.277593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10211cc8300 of size 512
2019-01-11 00:29:57.277610: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8500 of size 512
2019-01-11 00:29:57.277628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8700 of size 512
2019-01-11 00:29:57.277654: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8900 of size 1024
2019-01-11 00:29:57.277676: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8d00 of size 1024
2019-01-11 00:29:57.277701: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc9100 of size 1024
2019-01-11 00:29:57.277722: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc9500 of size 1024
2019-01-11 00:29:57.277748: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc9900 of size 1024
2019-01-11 00:29:57.277770: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10211cc9d00 of size 31232
2019-01-11 00:29:57.277798: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cd1700 of size 60823552
2019-01-11 00:29:57.277823: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102156d2f00 of size 58982400
2019-01-11 00:29:57.277851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10218f12f00 of size 58982400
2019-01-11 00:29:57.277878: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1021c752f00 of size 58982400
2019-01-11 00:29:57.277905: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1021ff92f00 of size 14745600
2019-01-11 00:29:57.277928: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10220da2f00 of size 14745600
2019-01-11 00:29:57.277955: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10221bb2f00 of size 14745600
2019-01-11 00:29:57.277982: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102229c2f00 of size 14745600
2019-01-11 00:29:57.278009: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102237d2f00 of size 14745600
2019-01-11 00:29:57.278037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102245e2f00 of size 3686400
2019-01-11 00:29:57.278065: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10224966f00 of size 59129856
2019-01-11 00:29:57.278092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102281caf00 of size 58982400
2019-01-11 00:29:57.278122: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022ba0af00 of size 7667712
2019-01-11 00:29:57.278149: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022c15af00 of size 14745600
2019-01-11 00:29:57.278177: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022cf6af00 of size 7372800
2019-01-11 00:29:57.278204: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022d672f00 of size 14893056
2019-01-11 00:29:57.278231: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022e4a6f00 of size 14745600
2019-01-11 00:29:57.278259: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022f2b6f00 of size 7962624
2019-01-11 00:29:57.278287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022fa4ef00 of size 7372800
2019-01-11 00:29:57.278315: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10230156f00 of size 1843200
2019-01-11 00:29:57.278343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10230318f00 of size 14893056
2019-01-11 00:29:57.278371: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1023114cf00 of size 14745600
2019-01-11 00:29:57.278394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10231f5cf00 of size 2695168
2019-01-11 00:29:57.278423: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102321eef00 of size 1843200
2019-01-11 00:29:57.278443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102323b0f00 of size 1024
2019-01-11 00:29:57.278473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323b1300 of size 512
2019-01-11 00:29:57.278493: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323b1500 of size 512
2019-01-11 00:29:57.278522: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323b1700 of size 1024
2019-01-11 00:29:57.278548: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323b1b00 of size 1024
2019-01-11 00:29:57.278576: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102323b1f00 of size 290816
2019-01-11 00:29:57.278603: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323f8f00 of size 2138112
2019-01-11 00:29:57.278630: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10232602f00 of size 1843200
2019-01-11 00:29:57.278656: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102327c4f00 of size 3686400
2019-01-11 00:29:57.278683: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10232b48f00 of size 3686400
2019-01-11 00:29:57.278709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10232eccf00 of size 1843200
2019-01-11 00:29:57.278736: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1023308ef00 of size 1843200
2019-01-11 00:29:57.278762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10233250f00 of size 7372800
2019-01-11 00:29:57.278788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10233958f00 of size 7372800
2019-01-11 00:29:57.278814: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10234060f00 of size 3686400
2019-01-11 00:29:57.278837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102343e4f00 of size 3686400
2019-01-11 00:29:57.278864: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10234768f00 of size 3686400
2019-01-11 00:29:57.278892: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10234aecf00 of size 7372800
2019-01-11 00:29:57.278919: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102351f4f00 of size 1179648
2019-01-11 00:29:57.278945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10235314f00 of size 7372800
2019-01-11 00:29:57.278972: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10235a1cf00 of size 936192
2019-01-11 00:29:57.278998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10235b01800 of size 2750208
2019-01-11 00:29:57.279024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10235da0f00 of size 3686400
2019-01-11 00:29:57.279051: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236124f00 of size 1843200
2019-01-11 00:29:57.279077: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102362e6f00 of size 589824
2019-01-11 00:29:57.279104: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236376f00 of size 1843200
2019-01-11 00:29:57.279130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236538f00 of size 1843200
2019-01-11 00:29:57.279158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102366faf00 of size 1843200
2019-01-11 00:29:57.279184: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102368bcf00 of size 3686400
2019-01-11 00:29:57.279211: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236c40f00 of size 3686400
2019-01-11 00:29:57.279238: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236fc4f00 of size 3686400
2019-01-11 00:29:57.279264: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10237348f00 of size 1843200
2019-01-11 00:29:57.279291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1023750af00 of size 9216000
2019-01-11 00:29:57.279318: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10237dd4f00 of size 3686400
2019-01-11 00:29:57.279344: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10238158f00 of size 3686400
2019-01-11 00:29:57.279373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102384dcf00 of size 4497120000
2019-01-11 00:29:57.279400: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x103445a6a00 of size 1930332416
2019-01-11 00:29:57.279424: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size:
2019-01-11 00:29:57.279448: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 46 Chunks of size 256 totalling 11.5KiB
2019-01-11 00:29:57.279469: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 19 Chunks of size 512 totalling 9.5KiB
2019-01-11 00:29:57.279486: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 768 totalling 768B
2019-01-11 00:29:57.279506: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 10 Chunks of size 1024 totalling 10.0KiB
2019-01-11 00:29:57.279523: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1280 totalling 2.5KiB
2019-01-11 00:29:57.279543: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 147456 totalling 432.0KiB
2019-01-11 00:29:57.279561: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 262144 totalling 256.0KiB
2019-01-11 00:29:57.279580: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-11 00:29:57.279597: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 589824 totalling 1.69MiB
2019-01-11 00:29:57.279617: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 921600 totalling 900.0KiB
2019-01-11 00:29:57.279642: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 921856 totalling 900.2KiB
2019-01-11 00:29:57.279671: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-11 00:29:57.279701: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 9 Chunks of size 1843200 totalling 15.82MiB
2019-01-11 00:29:57.279731: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2138112 totalling 2.04MiB
2019-01-11 00:29:57.279759: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2695168 totalling 2.57MiB
2019-01-11 00:29:57.279789: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2750208 totalling 2.62MiB
2019-01-11 00:29:57.279819: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 10 Chunks of size 3686400 totalling 35.16MiB
2019-01-11 00:29:57.279850: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 7372800 totalling 42.19MiB
2019-01-11 00:29:57.279879: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 7667712 totalling 7.31MiB
2019-01-11 00:29:57.279907: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 7962624 totalling 7.59MiB
2019-01-11 00:29:57.279937: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 8 Chunks of size 14745600 totalling 112.50MiB
2019-01-11 00:29:57.279968: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 14893056 totalling 28.41MiB
2019-01-11 00:29:57.279997: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 58982400 totalling 281.25MiB
2019-01-11 00:29:57.280028: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 59129856 totalling 56.39MiB
2019-01-11 00:29:57.280058: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 60823552 totalling 58.01MiB
2019-01-11 00:29:57.280088: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 62613248 totalling 59.71MiB
2019-01-11 00:29:57.280115: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4497120000 totalling 4.19GiB
2019-01-11 00:29:57.280143: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 4.89GiB
2019-01-11 00:29:57.280175: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 7202206516
InUse: 5249080064
MaxInUse: 6401664768
NumAllocs: 531
MaxAllocSize: 4497120000

2019-01-11 00:29:57.280248: W tensorflow/core/common_runtime/bfc_allocator.cc:279] *************************************************************************x__________________________
2019-01-11 00:29:57.280303: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[3,30,40,30,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1361, in _do_call
return fn(*args)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _run_fn
target_list, status, run_metadata)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,30,40,30,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: superpoint/train_tower0/gradients/superpoint/train_tower0/mul_3_grad/mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/Reshape_6, superpoint/train_tower0/gradients/superpoint/train_tower0/Sum_grad/Tile)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: superpoint/train_tower0/gradients/AddN_17/_427 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2157_superpoint/train_tower0/gradients/AddN_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "experiment.py", line 154, in
args.func(config, output_dir, args)
File "experiment.py", line 92, in _cli_train
train(config, config['train_iter'], output_dir)
File "experiment.py", line 33, in train
keep_checkpoints=config.get('keep_checkpoints', 1))
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 313, in train
options=options, run_metadata=run_metadata)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1137, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1355, in _do_run
options, run_metadata)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1374, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,30,40,30,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: superpoint/train_tower0/gradients/superpoint/train_tower0/mul_3_grad/mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/Reshape_6, superpoint/train_tower0/gradients/superpoint/train_tower0/Sum_grad/Tile)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: superpoint/train_tower0/gradients/AddN_17/_427 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2157_superpoint/train_tower0/gradients/AddN_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'superpoint/train_tower0/gradients/superpoint/train_tower0/mul_3_grad/mul_1', defined at:
File "experiment.py", line 154, in
args.func(config, output_dir, args)
File "experiment.py", line 92, in _cli_train
train(config, config['train_iter'], output_dir)
File "experiment.py", line 27, in train
with _init_graph(config) as net:
File "/opt/python3.6/lib/python3.6/contextlib.py", line 82, in enter
return next(self.gen)
File "experiment.py", line 77, in _init_graph
n_gpus=n_gpus, **config['model'])
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 122, in init
self._build_graph()
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 264, in _build_graph
self._train_graph(data)
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 188, in _train_graph
data, Mode.TRAIN, self.config['batch_size'])
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 164, in _gpu_tower
grad = tf.gradients(loss, model_params)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 611, in gradients
lambda: grad_fn(op, *out_grads))
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 377, in _MaybeCompile
return grad_fn() # Exit early
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 611, in
lambda: grad_fn(op, *out_grads))
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 798, in _MulGrad
array_ops.reshape(math_ops.reduce_sum(x * grad, ry), sy))
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 934, in binary_op_wrapper
return func(x, y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1161, in _mul_dispatch
return gen_math_ops._mul(x, y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2789, in _mul
"Mul", x=x, y=y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
op_def=op_def)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

...which was originally created as op 'superpoint/train_tower0/mul_3', defined at:
File "experiment.py", line 154, in
args.func(config, output_dir, args)
[elided 6 identical lines from previous traceback]
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 188, in _train_graph
data, Mode.TRAIN, self.config['batch_size'])
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 159, in _gpu_tower
loss = self._loss(net_outputs, shards[i], **self.config)
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/super_point.py", line 82, in _loss
valid_mask=inputs['warped']['valid_mask'], **config)
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/utils.py", line 103, in descriptor_loss
dot_product_desc = tf.reduce_sum(descriptors * warped_descriptors, -1)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 934, in binary_op_wrapper
return func(x, y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1161, in _mul_dispatch
return gen_math_ops._mul(x, y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2789, in _mul
"Mul", x=x, y=y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
op_def=op_def)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3,30,40,30,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: superpoint/train_tower0/gradients/superpoint/train_tower0/mul_3_grad/mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/Reshape_6, superpoint/train_tower0/gradients/superpoint/train_tower0/Sum_grad/Tile)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: superpoint/train_tower0/gradients/AddN_17/_427 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2157_superpoint/train_tower0/gradients/AddN_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

AssertionError

@rpautrat
When i run "python3 experiment.py train configs/magic-point_coco_train.yaml magic-point_coco", I got the AssertionError like picture (2). And i found that there does not exist the folder of mp_synth-v10_ha2-100 under /home/ivip/lidongjiang/tmp2/outputs. Under the folder of /home/ivip/lidongjiang/tmp2/outputs/ is .npz files created by step2) like the picture(1). Does this mean that i need to repeat the step2) ? it does cost much time. Can you give me some advise?
picture(1)

picture(2)

about super point train

did you train descriptors? it seems hard to converge to a good descriptor feature map as in paper.

if you trained the whole pipeline successfully, can you tell what's the key to train descriptor well? I tried many times while get poor des performance

value of precision and recall are very low when training magicpoint on coco dataset

hello,
i train magicpoint on coco dataset, but the value of precision and recall are very low

is it correct ?

And i train the magicpoint using pretrained model trainded on synthetic_shapes dataset.

Number of training images

Hi , i wonder how many images you use when you train Magicpoint with the Synthetic images and MS COCO. images

Following the config file (magic-point_shapes.yaml), 'train_iter' is 200,000 and 'batch_size' is 64.
Then did you use 12,800,000 synthetic images? or 12,800,000*NUM_GPUs?(How many Gpus you use?)

In 'magic-point_coco_export.yaml', 'train_iter' is 18000 and 'batch_size' is 32.
Then Number of total images is 576,000. Number of 'COCO 2014 train' images is 82,783. 576,000 can't divide by 82,783.

And Is the config file of the pretrained model the same as the uploaded file in Github?

Could not find opencv-python-contrib, there may be an error in the contents of requirements.txt?

hello, I encounter a error when I install the required dependency through type: pipe install -r requirement.txt. The error display : Could not find a version that satisfies the requirement opencv-python-contrib (from versions: ) No matching distribution found for opencv-python-contrib.

Then, I tried to install opencv-python-contrib by type: pip install opencv-contrib-python, and the installation is successful. But the requirements.txt is written opencv-python-contrib instead of opencv-contrib-python, so I guess the requirements.txt is wrong here.
Is that right?

Cholesky decomposition error

when I try to use the cmd "python experiment.py train configs/magic-point_shapes.yaml magic-point_synth" on my gtx1070 gpu,Iter 0 is executed.But,and then no less than 1000 iterations,error prompt pops up and says the Cholesky decomposition was not successful.I don't know why and I wonder if the code can run on a GTX1070 GPU.Can you give me any suggestions?

Non GPU version

Is it possible to run without GPU. How to run only on CPU?

How to produce descriptors for images?

Hi, thank you for sharing this repo.
I have trained a superpoint model based on your repo, and I can produce key points by using the model, the command I used is "python export_detections.py configs/super-point_coco_export.yaml superpoint_coco --pred_only --batch_size=5 --export_name=super-point-export". The key points produced by the trained model looks reasonable, but I failed to produce descriptors for input images.
Could you please provide some hints on this? Thank you very much!

Unexpected results with pretrained model

Hello, thank you very much for this repo. Are there examples available for using the pretrained model pred_magic-point_17-06-2018? My current attempts produce rather unexpected results on the COCO dataset, where most the interest points seem to be concentrated at the corners.

After following the instructions from #18 , removing line 70 if not ('name' in d): to include the image on the npz-file, I ran the following command to export the interest points:

python3.6 ./superpoint/export_detections.py superpoint/configs/magic-point_coco_export.yaml magic-point_17-06-2018 --export_name pred_magic-point_17-06-2018

Lastly I've used this script to visualize the interest-points from the first 5 images from the coco dataset:

#!/usr/bin/python3.6
import cv2
import numpy as np
import os
import glob

from superpoint.settings import EXPER_PATH

def draw_keypoints(img, corners, color=(0, 255, 0), radius=3, s=3):
    img = np.repeat(cv2.resize(img, None, fx=s, fy=s)[..., np.newaxis], 3, -1)
    for c in np.stack(corners).T:
        cv2.circle(img, tuple(s*np.flip(c, 0)), radius, color, thickness=-1)
    return img
    
def select_top_k(prob, thresh=0, num=300):    
    pts = np.where(prob > thresh)
    idx = np.argsort(prob[pts])[::-1][:num]
    pts = (pts[0][idx], pts[1][idx])
    return pts
    
experiment = 'pred_magic-point_17-06-2018'
files = glob.glob("{}/*.npz".format(os.path.join(EXPER_PATH, "outputs", experiment)))

for i, path in enumerate(files[:5]):
    d = np.load(path)
    points1 = select_top_k(d['prob'], thresh=0)
    im1 = draw_keypoints(d['image'][..., 0], points1, (0, 255, 0))
    cv2.imwrite("interest_points_{}.png".format(i), im1)

But the interest points seem to be completely off. What am I doing wrong here?

Experiments parameters missing like harris_coco-repeat

In the descriptors_evaluation_on_hpatches.ipynb

experiments = [
'mp_synth-v6_photo-hom-aug_ha2-100-3-old_coco-repeat', 'harris_coco-repeat']
confidence_thresholds = [0.015, 0]

file not found. How to generate the harris_coco_repeat?

retrian issue

Thanks for u fantastic code.
There are somethings trouble me，this code never use the trained model from COCO or synthetic dataset，can u share the reason?
NEXT problem is why we can not use the offical superpoint trained model to train our own model，can u share this code and experience

how to

killed

Killed during train of using superpoint_hpatches.yaml

After 25000 iteration the training is killed. How to complete the iterations so that it saves some file in checkpoint.

python experiment.py train configs/superpoint_coco.yaml superpoint_coco

detector_evaluation_magic_point error

run PR curve:
KeyError: 'prob is not a file in the archive'
run Localization Error:
KeyError: 'keypoint_map is not a file in the archive'
run Visualization:
KeyError:'image is not a file in the archive'

where is the problem?

TypeError: invalid file: PosixPath

@rpautrat
When I run "python3 experiment.py train configs/magic-point_shapes.yaml magic-point_synth", i got the following error. the version of python is 3.5.2. is it the reason? I can not change this python version, because many other people still need to use. How should do?

The problem of Training of SuperPoint on MS-COCO

@rpautrat
When i run the step6) "python3 experiment.py train configs/superpoint_coco.yaml superpoint_coco", I got the following statements, so how should i do? is there anything wrong? I need some advice, Thank you very much.

$ python3 experiment.py train configs/superpoint_coco.yaml superpoint_coco
[01/11/2019 00:29:25 INFO] Running command TRAIN
[01/11/2019 00:29:25 INFO] Number of GPUs detected: 1
[01/11/2019 00:29:28 INFO] Caching data, fist access will take some time.
[01/11/2019 00:29:29 INFO] Caching data, fist access will take some time.
[01/11/2019 00:29:30 INFO] Caching data, fist access will take some time.
2019-01-11 00:29:31.373688: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-01-11 00:29:31.441013: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-01-11 00:29:31.441328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.695
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 7.11GiB
2019-01-11 00:29:31.441340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2019-01-11 00:29:31.610840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6868 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
[01/11/2019 00:29:31 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:31 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:32 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
[01/11/2019 00:29:33 INFO] Scale of 0 disables regularizer.
2019-01-11 00:29:33.541473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2019-01-11 00:29:33.541570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 135 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
[01/11/2019 00:29:41 INFO] Start training
2019-01-11 00:29:45.965791: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x7f8dc016dbb0
2019-01-11 00:29:57.274840: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.12GiB. Current allocation summary follows.
2019-01-11 00:29:57.274945: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): Total Chunks: 50, Chunks in use: 46. 12.5KiB allocated for chunks. 11.5KiB in use in bin. 5.1KiB client-requested in use in bin.
2019-01-11 00:29:57.274977: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): Total Chunks: 26, Chunks in use: 20. 13.5KiB allocated for chunks. 10.2KiB in use in bin. 10.0KiB client-requested in use in bin.
2019-01-11 00:29:57.275005: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): Total Chunks: 17, Chunks in use: 12. 17.8KiB allocated for chunks. 12.5KiB in use in bin. 12.0KiB client-requested in use in bin.
2019-01-11 00:29:57.275037: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275060: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275093: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): Total Chunks: 1, Chunks in use: 0. 15.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275118: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): Total Chunks: 1, Chunks in use: 0. 30.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275137: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): Total Chunks: 1, Chunks in use: 0. 56.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275157: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): Total Chunks: 1, Chunks in use: 0. 69.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275180: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): Total Chunks: 3, Chunks in use: 3. 432.0KiB allocated for chunks. 432.0KiB in use in bin. 432.0KiB client-requested in use in bin.
2019-01-11 00:29:57.275201: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): Total Chunks: 3, Chunks in use: 2. 828.0KiB allocated for chunks. 544.0KiB in use in bin. 544.0KiB client-requested in use in bin.
2019-01-11 00:29:57.275221: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): Total Chunks: 7, Chunks in use: 5. 4.90MiB allocated for chunks. 3.45MiB in use in bin. 3.45MiB client-requested in use in bin.
2019-01-11 00:29:57.275245: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): Total Chunks: 13, Chunks in use: 10. 20.95MiB allocated for chunks. 16.95MiB in use in bin. 16.95MiB client-requested in use in bin.
2019-01-11 00:29:57.275267: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): Total Chunks: 15, Chunks in use: 13. 49.42MiB allocated for chunks. 42.39MiB in use in bin. 40.43MiB client-requested in use in bin.
2019-01-11 00:29:57.275289: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): Total Chunks: 8, Chunks in use: 8. 57.09MiB allocated for chunks. 57.09MiB in use in bin. 56.25MiB client-requested in use in bin.
2019-01-11 00:29:57.275312: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): Total Chunks: 11, Chunks in use: 10. 149.70MiB allocated for chunks. 140.91MiB in use in bin. 140.62MiB client-requested in use in bin.
2019-01-11 00:29:57.275331: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275354: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): Total Chunks: 8, Chunks in use: 8. 455.36MiB allocated for chunks. 455.36MiB in use in bin. 450.00MiB client-requested in use in bin.
2019-01-11 00:29:57.275372: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275389: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-11 00:29:57.275409: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): Total Chunks: 2, Chunks in use: 1. 5.99GiB allocated for chunks. 4.19GiB in use in bin. 4.12GiB client-requested in use in bin.
2019-01-11 00:29:57.275429: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 4.12GiB was 256.00MiB, Chunk State:
2019-01-11 00:29:57.275459: I tensorflow/core/common_runtime/bfc_allocator.cc:652] Size: 1.80GiB | Requested Size: 17.16MiB | in_use: 0, prev: Size: 4.19GiB | Requested Size: 4.12GiB | in_use: 1
2019-01-11 00:29:57.275481: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200000 of size 1280
2019-01-11 00:29:57.275498: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200500 of size 1280
2019-01-11 00:29:57.275522: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200a00 of size 256
2019-01-11 00:29:57.275545: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200b00 of size 256
2019-01-11 00:29:57.275568: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200c00 of size 256
2019-01-11 00:29:57.275592: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200d00 of size 256
2019-01-11 00:29:57.275615: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200e00 of size 256
2019-01-11 00:29:57.275638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a200f00 of size 256
2019-01-11 00:29:57.275660: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201000 of size 256
2019-01-11 00:29:57.275676: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201100 of size 256
2019-01-11 00:29:57.275690: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201200 of size 256
2019-01-11 00:29:57.275710: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201300 of size 256
2019-01-11 00:29:57.275730: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201400 of size 256
2019-01-11 00:29:57.275745: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201500 of size 256
2019-01-11 00:29:57.275760: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201600 of size 256
2019-01-11 00:29:57.275777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201700 of size 256
2019-01-11 00:29:57.275812: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201800 of size 256
2019-01-11 00:29:57.275831: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201900 of size 256
2019-01-11 00:29:57.275846: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201a00 of size 256
2019-01-11 00:29:57.275866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201b00 of size 256
2019-01-11 00:29:57.275889: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201c00 of size 256
2019-01-11 00:29:57.275914: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201d00 of size 256
2019-01-11 00:29:57.275939: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201e00 of size 256
2019-01-11 00:29:57.275963: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a201f00 of size 256
2019-01-11 00:29:57.275995: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202000 of size 256
2019-01-11 00:29:57.276012: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202100 of size 256
2019-01-11 00:29:57.276035: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202200 of size 256
2019-01-11 00:29:57.276059: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202300 of size 256
2019-01-11 00:29:57.276088: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a202400 of size 589824
2019-01-11 00:29:57.276106: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a292400 of size 512
2019-01-11 00:29:57.276121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a292600 of size 512
2019-01-11 00:29:57.276136: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a292800 of size 589824
2019-01-11 00:29:57.276153: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a322800 of size 512
2019-01-11 00:29:57.276174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a322a00 of size 512
2019-01-11 00:29:57.276215: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a322c00 of size 512
2019-01-11 00:29:57.276232: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a322e00 of size 512
2019-01-11 00:29:57.276247: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a323000 of size 589824
2019-01-11 00:29:57.276262: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3b3000 of size 512
2019-01-11 00:29:57.276277: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3b3200 of size 512
2019-01-11 00:29:57.276292: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a3b3400 of size 768
2019-01-11 00:29:57.276307: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3b3700 of size 512
2019-01-11 00:29:57.276322: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3b3900 of size 294912
2019-01-11 00:29:57.276337: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a3fb900 of size 256
2019-01-11 00:29:57.276352: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3fba00 of size 512
2019-01-11 00:29:57.276367: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a3fbc00 of size 512
2019-01-11 00:29:57.276382: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a3fbe00 of size 262144
2019-01-11 00:29:57.276398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43be00 of size 1024
2019-01-11 00:29:57.276421: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a43c200 of size 1024
2019-01-11 00:29:57.276442: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43c600 of size 256
2019-01-11 00:29:57.276467: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43c700 of size 256
2019-01-11 00:29:57.276499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43c800 of size 256
2019-01-11 00:29:57.276515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43c900 of size 256
2019-01-11 00:29:57.276537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43ca00 of size 256
2019-01-11 00:29:57.276561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43cb00 of size 512
2019-01-11 00:29:57.276587: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43cd00 of size 768
2019-01-11 00:29:57.276612: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a43d000 of size 1024
2019-01-11 00:29:57.276638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43d400 of size 256
2019-01-11 00:29:57.276658: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a43d500 of size 256
2019-01-11 00:29:57.276684: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a43d600 of size 147456
2019-01-11 00:29:57.276711: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a461600 of size 256
2019-01-11 00:29:57.276731: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a461700 of size 1179648
2019-01-11 00:29:57.276756: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a581700 of size 1280
2019-01-11 00:29:57.276785: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a581c00 of size 1024
2019-01-11 00:29:57.276807: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a582000 of size 1024
2019-01-11 00:29:57.276835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a582400 of size 256
2019-01-11 00:29:57.276854: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a582500 of size 1179904
2019-01-11 00:29:57.276879: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6a2600 of size 256
2019-01-11 00:29:57.276906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a6a2700 of size 71168
2019-01-11 00:29:57.276927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6b3d00 of size 147456
2019-01-11 00:29:57.276954: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a6d7d00 of size 256
2019-01-11 00:29:57.276980: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6d7e00 of size 256
2019-01-11 00:29:57.277006: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6d7f00 of size 256
2019-01-11 00:29:57.277026: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6d8000 of size 147456
2019-01-11 00:29:57.277057: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a6fc000 of size 921600
2019-01-11 00:29:57.277078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a7dd000 of size 921856
2019-01-11 00:29:57.277094: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1020a8be100 of size 58112
2019-01-11 00:29:57.277120: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020a8cc400 of size 62613248
2019-01-11 00:29:57.277140: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1020e482b00 of size 58982400
2019-01-11 00:29:57.277165: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10211cc2b00 of size 15360
2019-01-11 00:29:57.277191: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6700 of size 256
2019-01-11 00:29:57.277208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6800 of size 256
2019-01-11 00:29:57.277223: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6900 of size 256
2019-01-11 00:29:57.277243: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6a00 of size 256
2019-01-11 00:29:57.277258: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6b00 of size 256
2019-01-11 00:29:57.277273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6c00 of size 256
2019-01-11 00:29:57.277298: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6d00 of size 256
2019-01-11 00:29:57.277324: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10211cc6e00 of size 256
2019-01-11 00:29:57.277343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc6f00 of size 256
2019-01-11 00:29:57.277372: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7000 of size 256
2019-01-11 00:29:57.277400: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7100 of size 512
2019-01-11 00:29:57.277418: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7300 of size 512
2019-01-11 00:29:57.277436: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7500 of size 512
2019-01-11 00:29:57.277467: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7700 of size 512
2019-01-11 00:29:57.277494: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7900 of size 512
2019-01-11 00:29:57.277515: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7b00 of size 1024
2019-01-11 00:29:57.277542: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc7f00 of size 512
2019-01-11 00:29:57.277567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8100 of size 512
2019-01-11 00:29:57.277593: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10211cc8300 of size 512
2019-01-11 00:29:57.277610: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8500 of size 512
2019-01-11 00:29:57.277628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8700 of size 512
2019-01-11 00:29:57.277654: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8900 of size 1024
2019-01-11 00:29:57.277676: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc8d00 of size 1024
2019-01-11 00:29:57.277701: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc9100 of size 1024
2019-01-11 00:29:57.277722: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc9500 of size 1024
2019-01-11 00:29:57.277748: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cc9900 of size 1024
2019-01-11 00:29:57.277770: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10211cc9d00 of size 31232
2019-01-11 00:29:57.277798: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10211cd1700 of size 60823552
2019-01-11 00:29:57.277823: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102156d2f00 of size 58982400
2019-01-11 00:29:57.277851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10218f12f00 of size 58982400
2019-01-11 00:29:57.277878: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1021c752f00 of size 58982400
2019-01-11 00:29:57.277905: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1021ff92f00 of size 14745600
2019-01-11 00:29:57.277928: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10220da2f00 of size 14745600
2019-01-11 00:29:57.277955: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10221bb2f00 of size 14745600
2019-01-11 00:29:57.277982: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102229c2f00 of size 14745600
2019-01-11 00:29:57.278009: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102237d2f00 of size 14745600
2019-01-11 00:29:57.278037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102245e2f00 of size 3686400
2019-01-11 00:29:57.278065: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10224966f00 of size 59129856
2019-01-11 00:29:57.278092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102281caf00 of size 58982400
2019-01-11 00:29:57.278122: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022ba0af00 of size 7667712
2019-01-11 00:29:57.278149: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022c15af00 of size 14745600
2019-01-11 00:29:57.278177: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022cf6af00 of size 7372800
2019-01-11 00:29:57.278204: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022d672f00 of size 14893056
2019-01-11 00:29:57.278231: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022e4a6f00 of size 14745600
2019-01-11 00:29:57.278259: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022f2b6f00 of size 7962624
2019-01-11 00:29:57.278287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1022fa4ef00 of size 7372800
2019-01-11 00:29:57.278315: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10230156f00 of size 1843200
2019-01-11 00:29:57.278343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10230318f00 of size 14893056
2019-01-11 00:29:57.278371: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1023114cf00 of size 14745600
2019-01-11 00:29:57.278394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10231f5cf00 of size 2695168
2019-01-11 00:29:57.278423: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102321eef00 of size 1843200
2019-01-11 00:29:57.278443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102323b0f00 of size 1024
2019-01-11 00:29:57.278473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323b1300 of size 512
2019-01-11 00:29:57.278493: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323b1500 of size 512
2019-01-11 00:29:57.278522: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323b1700 of size 1024
2019-01-11 00:29:57.278548: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323b1b00 of size 1024
2019-01-11 00:29:57.278576: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102323b1f00 of size 290816
2019-01-11 00:29:57.278603: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102323f8f00 of size 2138112
2019-01-11 00:29:57.278630: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10232602f00 of size 1843200
2019-01-11 00:29:57.278656: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102327c4f00 of size 3686400
2019-01-11 00:29:57.278683: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10232b48f00 of size 3686400
2019-01-11 00:29:57.278709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10232eccf00 of size 1843200
2019-01-11 00:29:57.278736: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x1023308ef00 of size 1843200
2019-01-11 00:29:57.278762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10233250f00 of size 7372800
2019-01-11 00:29:57.278788: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10233958f00 of size 7372800
2019-01-11 00:29:57.278814: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10234060f00 of size 3686400
2019-01-11 00:29:57.278837: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102343e4f00 of size 3686400
2019-01-11 00:29:57.278864: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10234768f00 of size 3686400
2019-01-11 00:29:57.278892: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10234aecf00 of size 7372800
2019-01-11 00:29:57.278919: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102351f4f00 of size 1179648
2019-01-11 00:29:57.278945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10235314f00 of size 7372800
2019-01-11 00:29:57.278972: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x10235a1cf00 of size 936192
2019-01-11 00:29:57.278998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10235b01800 of size 2750208
2019-01-11 00:29:57.279024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10235da0f00 of size 3686400
2019-01-11 00:29:57.279051: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236124f00 of size 1843200
2019-01-11 00:29:57.279077: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102362e6f00 of size 589824
2019-01-11 00:29:57.279104: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236376f00 of size 1843200
2019-01-11 00:29:57.279130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236538f00 of size 1843200
2019-01-11 00:29:57.279158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x102366faf00 of size 1843200
2019-01-11 00:29:57.279184: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102368bcf00 of size 3686400
2019-01-11 00:29:57.279211: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236c40f00 of size 3686400
2019-01-11 00:29:57.279238: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10236fc4f00 of size 3686400
2019-01-11 00:29:57.279264: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10237348f00 of size 1843200
2019-01-11 00:29:57.279291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x1023750af00 of size 9216000
2019-01-11 00:29:57.279318: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10237dd4f00 of size 3686400
2019-01-11 00:29:57.279344: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x10238158f00 of size 3686400
2019-01-11 00:29:57.279373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x102384dcf00 of size 4497120000
2019-01-11 00:29:57.279400: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x103445a6a00 of size 1930332416
2019-01-11 00:29:57.279424: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size:
2019-01-11 00:29:57.279448: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 46 Chunks of size 256 totalling 11.5KiB
2019-01-11 00:29:57.279469: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 19 Chunks of size 512 totalling 9.5KiB
2019-01-11 00:29:57.279486: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 768 totalling 768B
2019-01-11 00:29:57.279506: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 10 Chunks of size 1024 totalling 10.0KiB
2019-01-11 00:29:57.279523: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1280 totalling 2.5KiB
2019-01-11 00:29:57.279543: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 147456 totalling 432.0KiB
2019-01-11 00:29:57.279561: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 262144 totalling 256.0KiB
2019-01-11 00:29:57.279580: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-11 00:29:57.279597: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 589824 totalling 1.69MiB
2019-01-11 00:29:57.279617: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 921600 totalling 900.0KiB
2019-01-11 00:29:57.279642: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 921856 totalling 900.2KiB
2019-01-11 00:29:57.279671: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-11 00:29:57.279701: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 9 Chunks of size 1843200 totalling 15.82MiB
2019-01-11 00:29:57.279731: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2138112 totalling 2.04MiB
2019-01-11 00:29:57.279759: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2695168 totalling 2.57MiB
2019-01-11 00:29:57.279789: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2750208 totalling 2.62MiB
2019-01-11 00:29:57.279819: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 10 Chunks of size 3686400 totalling 35.16MiB
2019-01-11 00:29:57.279850: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 7372800 totalling 42.19MiB
2019-01-11 00:29:57.279879: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 7667712 totalling 7.31MiB
2019-01-11 00:29:57.279907: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 7962624 totalling 7.59MiB
2019-01-11 00:29:57.279937: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 8 Chunks of size 14745600 totalling 112.50MiB
2019-01-11 00:29:57.279968: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 14893056 totalling 28.41MiB
2019-01-11 00:29:57.279997: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 58982400 totalling 281.25MiB
2019-01-11 00:29:57.280028: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 59129856 totalling 56.39MiB
2019-01-11 00:29:57.280058: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 60823552 totalling 58.01MiB
2019-01-11 00:29:57.280088: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 62613248 totalling 59.71MiB
2019-01-11 00:29:57.280115: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4497120000 totalling 4.19GiB
2019-01-11 00:29:57.280143: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 4.89GiB
2019-01-11 00:29:57.280175: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 7202206516
InUse: 5249080064
MaxInUse: 6401664768
NumAllocs: 531
MaxAllocSize: 4497120000

2019-01-11 00:29:57.280248: W tensorflow/core/common_runtime/bfc_allocator.cc:279] *************************************************************************x__________________________
2019-01-11 00:29:57.280303: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[3,30,40,30,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1361, in _do_call
return fn(*args)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _run_fn
target_list, status, run_metadata)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,30,40,30,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: superpoint/train_tower0/gradients/superpoint/train_tower0/mul_3_grad/mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/Reshape_6, superpoint/train_tower0/gradients/superpoint/train_tower0/Sum_grad/Tile)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: superpoint/train_tower0/gradients/AddN_17/_427 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2157_superpoint/train_tower0/gradients/AddN_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "experiment.py", line 154, in
args.func(config, output_dir, args)
File "experiment.py", line 92, in _cli_train
train(config, config['train_iter'], output_dir)
File "experiment.py", line 33, in train
keep_checkpoints=config.get('keep_checkpoints', 1))
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 313, in train
options=options, run_metadata=run_metadata)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1137, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1355, in _do_run
options, run_metadata)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1374, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,30,40,30,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: superpoint/train_tower0/gradients/superpoint/train_tower0/mul_3_grad/mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/Reshape_6, superpoint/train_tower0/gradients/superpoint/train_tower0/Sum_grad/Tile)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: superpoint/train_tower0/gradients/AddN_17/_427 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2157_superpoint/train_tower0/gradients/AddN_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'superpoint/train_tower0/gradients/superpoint/train_tower0/mul_3_grad/mul_1', defined at:
File "experiment.py", line 154, in
args.func(config, output_dir, args)
File "experiment.py", line 92, in _cli_train
train(config, config['train_iter'], output_dir)
File "experiment.py", line 27, in train
with _init_graph(config) as net:
File "/opt/python3.6/lib/python3.6/contextlib.py", line 82, in enter
return next(self.gen)
File "experiment.py", line 77, in _init_graph
n_gpus=n_gpus, **config['model'])
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 122, in init
self._build_graph()
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 264, in _build_graph
self._train_graph(data)
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 188, in _train_graph
data, Mode.TRAIN, self.config['batch_size'])
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 164, in _gpu_tower
grad = tf.gradients(loss, model_params)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 611, in gradients
lambda: grad_fn(op, *out_grads))
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 377, in _MaybeCompile
return grad_fn() # Exit early
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 611, in
lambda: grad_fn(op, *out_grads))
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 798, in _MulGrad
array_ops.reshape(math_ops.reduce_sum(x * grad, ry), sy))
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 934, in binary_op_wrapper
return func(x, y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1161, in _mul_dispatch
return gen_math_ops._mul(x, y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2789, in _mul
"Mul", x=x, y=y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
op_def=op_def)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

...which was originally created as op 'superpoint/train_tower0/mul_3', defined at:
File "experiment.py", line 154, in
args.func(config, output_dir, args)
[elided 6 identical lines from previous traceback]
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 188, in _train_graph
data, Mode.TRAIN, self.config['batch_size'])
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/base_model.py", line 159, in _gpu_tower
loss = self._loss(net_outputs, shards[i], **self.config)
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/super_point.py", line 82, in _loss
valid_mask=inputs['warped']['valid_mask'], **config)
File "/home/ivip/lidongjiang/SuperPoint/superpoint/models/utils.py", line 103, in descriptor_loss
dot_product_desc = tf.reduce_sum(descriptors * warped_descriptors, -1)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 934, in binary_op_wrapper
return func(x, y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 1161, in _mul_dispatch
return gen_math_ops._mul(x, y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2789, in _mul
"Mul", x=x, y=y, name=name)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
op_def=op_def)
File "/home/ivip/lidongjiang/SP-venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3,30,40,30,40,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: superpoint/train_tower0/gradients/superpoint/train_tower0/mul_3_grad/mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](superpoint/train_tower0/Reshape_6, superpoint/train_tower0/gradients/superpoint/train_tower0/Sum_grad/Tile)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: superpoint/train_tower0/gradients/AddN_17/_427 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2157_superpoint/train_tower0/gradients/AddN_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

data_format="NCHW"?

/usr/bin/python3.5 /home/li/SuperPoint/superpoint/experiment.py train configs/magic-point_shapes.yaml magic-point_synth
[12/30/2018 19:06:00 INFO] Running command TRAIN
[12/30/2018 19:06:00 INFO] Number of GPUs detected: 1
[12/30/2018 19:06:04 INFO] Extracting archive for primitive draw_lines.
[12/30/2018 19:06:09 INFO] Extracting archive for primitive draw_polygon.
[12/30/2018 19:06:18 INFO] Extracting archive for primitive draw_multiple_polygons.
[12/30/2018 19:06:36 INFO] Extracting archive for primitive draw_ellipses.
[12/30/2018 19:06:58 INFO] Extracting archive for primitive draw_star.
[12/30/2018 19:07:22 INFO] Extracting archive for primitive draw_checkerboard.
[12/30/2018 19:07:52 INFO] Extracting archive for primitive draw_stripes.
[12/30/2018 19:08:27 INFO] Extracting archive for primitive draw_cube.
[12/30/2018 19:09:04 INFO] Extracting archive for primitive gaussian_noise.
[12/30/2018 19:09:47 INFO] Caching data, fist access will take some time.
[12/30/2018 19:09:48 INFO] Caching data, fist access will take some time.
[12/30/2018 19:09:48 INFO] Caching data, fist access will take some time.
2018-12-30 19:09:48.298691: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-12-30 19:09:48.611304: E tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_NO_DEVICE
2018-12-30 19:09:48.613064: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (li-Alienware-15-R4): /proc/driver/nvidia/version does not exist
[12/30/2018 19:09:48 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:49 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:50 INFO] Scale of 0 disables regularizer.
[12/30/2018 19:09:53 INFO] Start training
2018-12-30 19:09:54.409166: E tensorflow/core/common_runtime/executor.cc:645] Executor failed to create kernel. Invalid argument: Default MaxPoolingOp only supports NHWC on device type CPU
[[Node: magicpoint/train_tower0/vgg/pool1/MaxPool = MaxPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="SAME", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Traceback (most recent call last):
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1361, in _do_call
return fn(*args)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1340, in _run_fn
target_list, status, run_metadata)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU
[[Node: magicpoint/train_tower0/vgg/pool1/MaxPool = MaxPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="SAME", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/li/SuperPoint/superpoint/experiment.py", line 160, in
args.func(config, output_dir, args)
File "/home/li/SuperPoint/superpoint/experiment.py", line 95, in _cli_train
train(config, config['train_iter'], output_dir)
File "/home/li/SuperPoint/superpoint/experiment.py", line 35, in train
keep_checkpoints=config.get('keep_checkpoints', 1))
File "../superpoint/models/base_model.py", line 320, in train
options=options, run_metadata=run_metadata)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 905, in run
run_metadata_ptr)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1137, in _run
feed_dict_tensor, options, run_metadata)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1355, in _do_run
options, run_metadata)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1374, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Default MaxPoolingOp only supports NHWC on device type CPU
[[Node: magicpoint/train_tower0/vgg/pool1/MaxPool = MaxPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="SAME", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'magicpoint/train_tower0/vgg/pool1/MaxPool', defined at:
File "/home/li/SuperPoint/superpoint/experiment.py", line 160, in
args.func(config, output_dir, args)
File "/home/li/SuperPoint/superpoint/experiment.py", line 95, in _cli_train
train(config, config['train_iter'], output_dir)
File "/home/li/SuperPoint/superpoint/experiment.py", line 29, in train
with _init_graph(config) as net:
File "/usr/lib/python3.5/contextlib.py", line 59, in enter
return next(self.gen)
File "/home/li/SuperPoint/superpoint/experiment.py", line 81, in _init_graph
n_gpus=n_gpus, **config['model'])
File "../superpoint/models/base_model.py", line 129, in init
self._build_graph()
File "../superpoint/models/base_model.py", line 271, in _build_graph
self._train_graph(data)
File "../superpoint/models/base_model.py", line 195, in _train_graph
data, Mode.TRAIN, self.config['batch_size'])
File "../superpoint/models/base_model.py", line 164, in _gpu_tower
net_outputs = self._model(shards[i], mode, **self.config)
File "../superpoint/models/magic_point.py", line 38, in _model
outputs = net(image)
File "../superpoint/models/magic_point.py", line 31, in net
features = vgg_backbone(image, **config)
File "../superpoint/models/backbones/vgg.py", line 28, in vgg_backbone
x = tfl.max_pooling2d(x, 2, 2, name='pool1', **params_pool)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/layers/pooling.py", line 433, in max_pooling2d
return layer.apply(inputs)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 809, in apply
return self.call(inputs, *args, **kwargs)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/layers/base.py", line 696, in call
outputs = self.call(inputs, *args, **kwargs)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/layers/pooling.py", line 277, in call
data_format=utils.convert_data_format(self.data_format, 4))
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 2124, in max_pool
name=name)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2981, in _max_pool
data_format=data_format, name=name)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
op_def=op_def)
File "/home/li/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1650, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Default MaxPoolingOp only supports NHWC on device type CPU
[[Node: magicpoint/train_tower0/vgg/pool1/MaxPool = MaxPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 2, 2], padding="SAME", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Process finished with exit code 1

repeatability on HPatches Dataset

Hi,
I computed repeatability score of pre-trained SuperPoint network using the repeatability code provided at https://github.com/rpautrat/SuperPoint/blob/master/superpoint/evaluations/detector_evaluation.py
I am getting the following results:

These results are not matching with the one reported in original paper. Can you help me to understand the results?

Restoring parameters problem

Ｉ have two problems:
(1) When i execute the program of the 4) step in the readme, as the following picture, when restoring about half parameters, it automatically stopped. is that normal?
(2) Another problem is that:
Is it means that what i need to do is to execute program instructed by step 1) 2) 3) 4) and 5) in the file of readme and i can get the right outcome? Because i do not know how to use other files in the file of configs like superpoint_coco.yaml and superpoint_hpatches.yaml. Those config files do not mentioned in the readme. I am a beginner and I am a little confused about those files.
export_descriptors.py and export_detections are also not mentioned in the readme.
I really need your help.

Descriptor training result in unreasonable matches and validation loss dramatically unstable

I'm trying to train superpoint net by keras (tensorflow backend) and COCO2014. During the training of descriptor I found the validation loss quite unstable, and the match drawed by opencv seems a mess. Is here anyone got a reasonable output of Desc net? Thanks. I can really use some advices.
Here is the outcome of my work:

TypeError: Sample_homography() got unexpected argument 'perspective_amplitude'

On the 5th Step while generating.

Python generate_coco_patches.py configs/coco_patches_generation.yaml
I got the perspective amplitude problem. How to solve this error?

about run step2 and step3 several times

Your work is quite great! The code is more like an artwork!
Well, I am wondering about whether the network's parameters will be restored in step3. It seems that when I run the step3, the network will be initialized and trained again. There seems no network restoring?
I haven't digged deeply in the codes, maybe I have a misunderstanding.

Training with very low precision and recall

Hello,
I run the training code as following,
python experiment.py train configs/magic-point_shapes.yaml magic-point_synth

the network starts to train and the loss is decreasing but the precision and recall are not changed.

2018-09-03 16:02:46.232410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
2018-09-03 16:02:46.232513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 292 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
[09/03/2018 16:02:48 INFO] Start training
[09/03/2018 16:03:00 INFO] Iter 0: loss 4.7797, precision 0.0006, recall 0.0615
[09/03/2018 16:05:05 INFO] Iter 1000: loss 1.5162, precision 0.0006, recall 0.0579
[09/03/2018 16:07:25 INFO] Iter 2000: loss 0.5852, precision 0.0005, recall 0.0541
[09/03/2018 16:09:44 INFO] Iter 3000: loss 0.2996, precision 0.0005, recall 0.0531
[09/03/2018 16:12:01 INFO] Iter 4000: loss 0.2295, precision 0.0006, recall 0.0445
[09/03/2018 16:14:14 INFO] Iter 5000: loss 0.2241, precision 0.0013, recall 0.0180
[09/03/2018 16:16:26 INFO] Iter 6000: loss 0.2034, precision 0.0016, recall 0.0063

How can I get the correct results?
Thank you!

running environment

hello, rpautrat, when i run this code on my linux server, i meet a question as follow:

i know that os.environ() functions has different attributes in windows and linux.
in SuperPoint-master/superpoint/datasets/synthetic_shape.py, this function is used to get a temp path, in windows environment, the function express should be os.environ['TEMP'], in linux, i have not find a suitable express, however, in this code file, the express is os.environ['TMPDIR'], is it wrong? i find the bug will appear whether in windows or in linux environment. what should i do if i want to run this code on my server that is Ubuntu 16.04.2. Thank you!

errors when Evaluating the repeatability on HPatches

hello, when evaluating the repeataability on HPatches, with the command 'python export_detections_repeatability.py configs/magic-point_repeatability.yaml magic-point_coco --export_name=magic-point_hpatches-repeatability-v' , i met some errors.
firstly, as follow picture:

it shows, the /datasets/patches_dataset.py, change the 40th row code, error, HPatches/v_fest doesn't have the files whose name start with 'H_1_2'. i see that dir, and i find the dir surely doesn't contain files whose name start with 'H_1_', but contains files whose name start with 'H_ref_', as follow picture:

so, i modified the /datasets/patches_dataset.py, change the 40th row code, from 'H_1_' to 'H_ref_', and run the command again, however i met a new error, as follow picture,

it shows, ValueError:could not convert string to float, and the 40th row error again, however, i see the code file, i don't find the convert from string to float, why this error appear, what should i do so that i could run this command successfully? thanks you

Notebook superpoint no module found

In detector_repeatability_coco.ipynb file
from superpoint.settings import EXPER_PATH

on execution using notebook
ModuleNotFoundError Traceback (most recent call last)
in ()
3 import matplotlib.pyplot as plt
4 from pathlib import Path
----> 5 from superpoint.settings import EXPER_PATH
6 import superpoint.evaluations.detector_evaluation as ev
7 from utils import plot_imgs

ModuleNotFoundError: No module named 'superpoint'
I tried to remove the word superpoint but it didnt work.

Loading pre-trained model and testing on image pairs

Hi, very interesting repo - thank you.
I'm trying to load the pre-trained model and use it to test inference on some sample images.
I'm having trouble doing so, if you could provide with a sample it would be great. here is what I tried:

with open("./configs/magic-point_coco_export.yaml", 'r') as f:
            config = yaml.load(f)
        dataset = get_dataset(config['data']['name'])(**config['data'])
        model = get_model(config['model']['name'])(
            data=dataset.get_tf_datasets(),
            n_gpus=1, **config['model'], pred=True)
        model.load("/home/omer/devel/git/ext_repos/SuperPoint/superpoint/saved_models/magic-point_17-06-2018")
        model.__enter__()

        model.__exit__()

Another attempt:

GRAPH_PB_PATH = '/home/omer/devel/git/ext_repos/SuperPoint/superpoint/saved_models/magic-point_17-06-2018/saved_model.pb'
       with tf.Session() as sess:
           print("load graph")
           with gfile.FastGFile(GRAPH_PB_PATH, 'rb') as f:
               graph_def = tf.GraphDef()
           graph_def.ParseFromString(f.read())
           sess.graph.as_default()
           tf.import_graph_def(graph_def, name='')
           graph_nodes = [n for n in graph_def.node]
           names = []
           for t in graph_nodes:
               names.append(t.name)
           print(names)

Using which descriptor the corner points are detected intially on the synthetic shapes.

In magic-point_shapes.yaml
if data.name is the name of the file. What does the primitive:all mean. How are the corner points detected on the synthetic shapes?

Export detections time

Hey @rpautrat, thanks again for the great implementation. I am currently in my second iteration of the training magic-point on the COCO dataset. It is taking forever though because the export_detections is running at 1.39 iterations per second. At this rate, it takes ~16 hrs to export all of the detections for the COCO train2014 dataset. Is this similar to what you were seeing? Or is there something wrong with my configuration? Thanks for your help!

Key Is_training not found in checkpoint

python export_detections_repeatability.py configs/magic-point_repeatability.yaml magic-point_synth

NotFoundError: Key is_training not found in checkpoint. How to solve this error?

Confirmation on the descriptor_evaluation flow

Dear Remi:

Thank you so much for building up this repo!

1. I downloaded the Hpatches full sequences from this site:
   https://github.com/hpatches/hpatches-dataset
   HPatches full sequences [1.3GB].

2. My command is:
   python export_descriptors.py configs/classical-descriptors.yaml orb_v --export_name=orb_hpatches-v

Then I use "descriptors_evaluation_on_hpatches_test.ipynb" to check the result.

(Note: I can produce the same result with you with "detector" evaluation + harris)

getting results of orb_hpatches-v: 0.10508474576271186 instead of 0.129 as you reported.

Would like to confirm if I have any incorrect settings? Thank you!

And this is my detailed config:

data:
name: 'patches_dataset'
dataset: 'hpatches' # 'hpatches' 'coco'
alteration: 'v' # 'i' 'v' 'all'
cache_in_memory: false
validation_size: 100
preprocessing:
resize: [480, 640] # False for coco
model:
name: 'classical_detectors_descriptors'
method: 'orb' # 'orb' 'sift'
batch_size: 50 # unused
learning_rate: 0.001 # unused
nms: 8
top_k: 1000
eval_iter: 600
seed: 1

errors about the Hpatches dataset groundtruth homography H_ref_1

Hello,I have met a problem when I try to map the feature points of the reference image to the warped image using the ground truth homography H_ref_X. I find that it will project to wrong feature points using the ground truth homography H_ref_X. For example，when I try to project the green point in the laptop of the first picture, the corresponding feature point (red) finded in the second picture using H_ref_X is not correct Obviously, the second point is even out of the image's boundary. Can you help me solve this problem? Thanks a lot.

'homography is not a file in the archive'

When I head over to descriptors_evaluation_on_hpatches.ipynb to proceed to the evaluation. I get an error

The experiment name is superpoint_hpatches-v just generated as the step 7 in README, but I only generate 10 .npz files. Does it matter?

NHWC problem:Default MaxPoolingOp only supports NHWC on device type CPU. how should i do?

Error in extracting Keypoints using homography adaptation

Hi, Thanks so much for the well written codebase for the Superpoint network.
While I was trying to replicate the results for COCO data, There was this error which occurred in the second part of the network(homography adaptation) to generate keypoints for the coco data.
Below is the attached log of the same.

Could you please let me know how to fix this issue? Thanks

$ python export_detections.py configs/magic-point_coco_export.yaml magic-point_synth --pred_only --batch_size=5 --export_name=magic-point_coco-export1
[01/03/2019 10:00:30 INFO] Number of GPUs detected: 1
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:39 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:40 INFO] Scale of 0 disables regularizer.
[01/03/2019 10:00:41 INFO] Restoring parameters from /home/ubuntu/tutorials/SuperPoint/b/magic-point_synth/model.ckpt-200000
0it [00:00, ?it/s]
20126it [1:26:50, 320131it [1:2620136it [1:26:52, 2014120120166i201736581it [2:38:03, 4.15it/s]2019-01-03 12:38:45.290810: E tensorflow/core/lib/jpeg/jpeg_mem.cc:307] Premature end of JPEG data. Stopped at line 56/396
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
return fn(*args)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid JPEG data or crop window, data size 32768
[[Node: DecodePng = DecodePngchannels=3, dtype=DT_UINT8, _device="/device:CPU:0"]]
[[Node: IteratorGetNext_2 = IteratorGetNextoutput_shapes=[[?,?,1], []], output_types=[DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "export_detections.py", line 53, in
data.append(next(test_set))
File "/home/ubuntu/tutorials/SuperPoint/superpoint/datasets/base_dataset.py", line 115, in _get_set_generator
yield self.sess.run(self.tf_next[set_name])
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid JPEG data or crop window, data size 32768
[[Node: DecodePng = DecodePngchannels=3, dtype=DT_UINT8, _device="/device:CPU:0"]]
[[Node: IteratorGetNext_2 = IteratorGetNextoutput_shapes=[[?,?,1], []], output_types=[DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

killed problem

Thanks for your fantastic code.
But I met a problem.
When I ran TMPDIR='/tmp' CUDA_VISIBLE_DEVICES='0' python experiment.py train configs/magic-point_shapes.yaml magic-point_synth
I got:
[03/16/2019 17:35:07 INFO] Start training
[1] 5974 killed TMPDIR='/tmp' CUDA_VISIBLE_DEVICES='0' python experiment.py train
How can I solve this ?
Thanks!

Is there has Pytorch version code?

Really thanks for your work, but in the paper said, ''All training is done using PyTorch''. And this opensource code is using Tensorflow. I wonder that if there has a Pytorch version code for SuperPoint?

Do we reuse pretrained parameter which learned the synthetic dataset?

Hi!

Thank you for your repo.

I am wondering we reuse the checkpoint which learned synthetic shape dataset when we train the model with MS COCO(step 3 part of Usage in README file). I tried to find restoring part of that, but I can't.

When I read the paper, authors pretrain the model about synthetic dataset and retrain about MS coco dataset which have pseudo keypoint labels detected by the model pretrained by synthetic dataset.

Thank you.

---

In base_model.py

TensorFlow do global initialization in the last part of _build_graph.

But there is no restoring parameter in train method.

def train(self, iterations, validation_interval=100, output_dir=None, profile=False,
          save_interval=None, checkpoint_path=None, keep_checkpoints=1):
    assert self.trainable, 'Model is not trainable.'
    assert 'training' in self.datasets, 'Training dataset is required.'
    if output_dir is not None:
        train_writer = tf.summary.FileWriter(output_dir)
    if not hasattr(self, 'saver'):
        with tf.device('/cpu:0'):
            self.saver = tf.train.Saver(save_relative_paths=True,
                                        max_to_keep=keep_checkpoints)
    if not self.graph.finalized:
        self.graph.finalize()
    if profile:
        options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
        run_metadata = tf.RunMetadata()
    else:
        options, run_metadata = None, None

    tf.logging.info('Start training')
    for i in range(iterations):
        loss, summaries, _ = self.sess.run(
                [self.loss, self.summaries, self.trainer],
                feed_dict={self.handle: self.dataset_handles['training']},
                options=options, run_metadata=run_metadata)

        if save_interval and checkpoint_path and (i+1) % save_interval == 0:
            self.save(checkpoint_path)
        if 'validation' in self.datasets and i % validation_interval == 0:
            metrics = self.evaluate('validation', mute=True)
            tf.logging.info(
                    'Iter {:4d}: loss {:.4f}'.format(i, loss) +
                    ''.join([', {} {:.4f}'.format(m, metrics[m]) for m in metrics]))

            if output_dir is not None:
                train_writer.add_summary(summaries, i)
                metrics_summaries = tf.Summary(value=[
                    tf.Summary.Value(tag=m, simple_value=v)
                    for m, v in metrics.items()])
                train_writer.add_summary(metrics_summaries, i)

                if profile and i != 0:
                    fetched_timeline = timeline.Timeline(run_metadata.step_stats)
                    chrome_trace = fetched_timeline.generate_chrome_trace_format()
                    with open(osp.join(output_dir,
                                       'profile_{}.json'.format(i)), 'w') as f:
                        f.write(chrome_trace)
    tf.logging.info('Training finished')

Question about dusbin.

Hi,
Thanks for your code, I have a small question, what is the benefit of using dustbin? Did you try to train the network with out dustbin?

Is it useful for document page corner detection?

Not an issue, but question: I wonder if I could train this on a synthetic dataset that container only corners like the ones from images of documents to be scanned with a camera.
Presumably it could learn their positions, right?

try to train, but cannot find some file like this

@rpautrat
python3 ./superpoint/experiment.py train ./superpoint/configs/magic-point_shapes.yaml magic-point_synth
[12/15/2018 13:59:30 INFO] Running command TRAIN
[12/15/2018 13:59:30 INFO] Number of GPUs detected: 1
[12/15/2018 13:59:32 INFO] Generating tarfile for primitive draw_lines.
Traceback (most recent call last):
File "./superpoint/experiment.py", line 149, in
args.func(config, output_dir, args)
File "./superpoint/experiment.py", line 87, in _cli_train
train(config, config['train_iter'], output_dir)
File "./superpoint/experiment.py", line 22, in train
with _init_graph(config) as net:
File "/usr/lib/python3.5/contextlib.py", line 59, in enter
return next(self.gen)
File "./superpoint/experiment.py", line 69, in _init_graph
dataset = get_dataset(config['data']['name'])(**config['data'])
File "/home/jie/project/SuperPoint-master/superpoint/datasets/base_dataset.py", line 102, in init
self.dataset = self._init_dataset(**self.config)
File "/home/jie/project/SuperPoint-master/superpoint/datasets/synthetic_shapes.py", line 126, in _init_dataset
self.dump_primitive_data(primitive, tar_path, config)
File "/home/jie/project/SuperPoint-master/superpoint/datasets/synthetic_shapes.py", line 100, in dump_primitive_data
tar = tarfile.open(tar_path, mode='w:gz')
File "/usr/lib/python3.5/tarfile.py", line 1575, in open
return func(name, filemode, fileobj, **kwargs)
File "/usr/lib/python3.5/tarfile.py", line 1622, in gzopen
fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
File "/usr/lib/python3.5/gzip.py", line 163, in init
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
TypeError: invalid file: PosixPath('/home/jie/project/SuperPoint-master/DATA_DIR/synthetic_shapes_v6/draw_lines.tar.gz')

KeyError: 'TMPDIR'

Hello! when I run the first command, a strange error happened as following. Can you tell me why ? Thank you very much !!!

error in trying to get mAP and Localization Error for detector_evaluation

hi ,when I try to follow the "detector_evaluation_magic_point.ipynb", I meet the trouble:

I don't know where the "keypoint_map" comes from.Does it mean the detection of the superpoint over standed image? Please help me,thanks a lot !

Is there a need to match the keypoint to at most one warped keypoint before computing the repeatability ?

Hello,
First of all, thank you for providing this code, especially the metrics code.
I was going through the repeatability computations (https://github.com/rpautrat/SuperPoint/blob/master/superpoint/evaluations/detector_evaluation.py) and more specifically in compute_repeatability.
I see that you compute the euclidean distance between the keypoints and count every pair with an euclidean distance less that distance_thresh. This means that if one keypoint match several others, let's say N, you count N match. And I was wondering if there is a need to do a kind of matching to ensure that there is only at most one match per keypoint ?

I am not sure of this because I have a lot trouble finding an explicit definition of repeatability that specifies how to do the match and there are a lot of versions. For example here, they specify that they do a matching (https://github.com/vlfeat/vlbenchmarks/blob/master/%2Bbenchmarks/RepeatabilityBenchmark.m) but I never saw it specified in a paper or in other versions of code as here (http://www.robots.ox.ac.uk/~vgg/research/affine/).

Thanks for your answer and if you know any resource that explicit the metrics definition (apart from Mikolajczyk papers), it would be help me a lot if you could share it please.

Thanks

Muti-GPU Training

Hi

I am trying to train the network with Multi-GPU. The training works fine with two GPUs but when I tried to train the network with more than two GPUs it gave an error. The log file is as below.

Command: python experiment.py train configs/magic-point_shapes.yaml magic-point_synth

[09/27/2018 10:22:23 INFO] Running command TRAIN
[09/27/2018 10:22:24 INFO] Number of GPUs detected: 4
[09/27/2018 10:22:26 INFO] Extracting archive for primitive draw_lines.
[09/27/2018 10:22:29 INFO] Extracting archive for primitive draw_polygon.
[09/27/2018 10:22:35 INFO] Extracting archive for primitive draw_multiple_polygons.
[09/27/2018 10:22:43 INFO] Extracting archive for primitive draw_ellipses.
[09/27/2018 10:22:55 INFO] Extracting archive for primitive draw_star.
[09/27/2018 10:23:10 INFO] Extracting archive for primitive draw_checkerboard.
[09/27/2018 10:23:30 INFO] Extracting archive for primitive draw_stripes.
[09/27/2018 10:23:53 INFO] Extracting archive for primitive draw_cube.
[09/27/2018 10:24:19 INFO] Extracting archive for primitive gaussian_noise.
[09/27/2018 10:24:50 INFO] Caching data, fist access will take some time.
[09/27/2018 10:24:51 INFO] Caching data, fist access will take some time.
[09/27/2018 10:24:51 INFO] Caching data, fist access will take some time.
2018-09-27 10:24:51.580540: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2018-09-27 10:24:52.087714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1b:00.0
totalMemory: 11.91GiB freeMemory: 11.68GiB
2018-09-27 10:24:52.395446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1c:00.0
totalMemory: 11.91GiB freeMemory: 11.74GiB
2018-09-27 10:24:52.683678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 2 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1d:00.0
totalMemory: 11.91GiB freeMemory: 11.74GiB
2018-09-27 10:24:52.990422: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 3 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1e:00.0
totalMemory: 11.91GiB freeMemory: 11.74GiB
2018-09-27 10:24:52.997493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1, 2, 3
2018-09-27 10:24:54.186225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-27 10:24:54.186264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0 1 2 3
2018-09-27 10:24:54.186270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N Y Y Y
2018-09-27 10:24:54.186273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1: Y N Y Y
2018-09-27 10:24:54.186276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 2: Y Y N Y
2018-09-27 10:24:54.186279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 3: Y Y Y N
2018-09-27 10:24:54.186907: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11305 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:1b:00.0, compute capability: 6.1)
2018-09-27 10:24:54.356548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 11363 MB memory) -> physical GPU (device: 1, name: TITAN Xp, pci bus id: 0000:1c:00.0, compute capability: 6.1)
2018-09-27 10:24:54.531258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 11363 MB memory) -> physical GPU (device: 2, name: TITAN Xp, pci bus id: 0000:1d:00.0, compute capability: 6.1)
2018-09-27 10:24:54.704548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 11363 MB memory) -> physical GPU (device: 3, name: TITAN Xp, pci bus id: 0000:1e:00.0, compute capability: 6.1)
[09/27/2018 10:24:55 INFO] Scale of 0 disables regularizer.
[09/27/2018 10:24:55 INFO] Scale of 0 disables regularizer.
[09/27/2018 10:24:55 INFO] Scale of 0 disables regularizer.
.
.
.
[09/27/2018 10:24:58 INFO] Scale of 0 disables regularizer.
2018-09-27 10:24:58.889130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1, 2, 3
2018-09-27 10:24:58.889303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-27 10:24:58.889312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0 1 2 3
2018-09-27 10:24:58.889317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N Y Y Y
2018-09-27 10:24:58.889321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1: Y N Y Y
2018-09-27 10:24:58.889325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 2: Y Y N Y
2018-09-27 10:24:58.889328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 3: Y Y Y N
2018-09-27 10:24:58.889819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11305 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:1b:00.0, compute capability: 6.1)
2018-09-27 10:24:58.889962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 11363 MB memory) -> physical GPU (device: 1, name: TITAN Xp, pci bus id: 0000:1c:00.0, compute capability: 6.1)
2018-09-27 10:24:58.890073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 11363 MB memory) -> physical GPU (device: 2, name: TITAN Xp, pci bus id: 0000:1d:00.0, compute capability: 6.1)
2018-09-27 10:24:58.890209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 11363 MB memory) -> physical GPU (device: 3, name: TITAN Xp, pci bus id: 0000:1e:00.0, compute capability: 6.1)
[09/27/2018 10:25:02 INFO] Start training
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
return fn(*args)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input shape axis 0 must equal 200, got shape [100,120,160]
[[Node: magicpoint/eval_data_sharding/unstack_3 = UnpackT=DT_INT32, axis=0, num=200, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: magicpoint/eval_tower2/map/while/box_nms/non_max_suppression/NonMaxSuppressionV3/_1553 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:2", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_1376_...pressionV3", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:2"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "experiment.py", line 151, in
args.func(config, output_dir, args)
File "experiment.py", line 89, in _cli_train
train(config, config['train_iter'], output_dir)
File "experiment.py", line 27, in train
keep_checkpoints=config.get('keep_checkpoints', 1))
File "/home/ubuntu/Downloads/SuperPoint_Test2/SuperPoint-master/superpoint/models/base_model.py", line 318, in train
metrics = self.evaluate('validation', mute=True)
File "/home/ubuntu/Downloads/SuperPoint_Test2/SuperPoint-master/superpoint/models/base_model.py", line 371, in evaluate
feed_dict={self.handle: self.dataset_handles[dataset]}))
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input shape axis 0 must equal 200, got shape [100,120,160]
[[Node: magicpoint/eval_data_sharding/unstack_3 = UnpackT=DT_INT32, axis=0, num=200, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: magicpoint/eval_tower2/map/while/box_nms/non_max_suppression/NonMaxSuppressionV3/_1553 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:2", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_1376_...pressionV3", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:2"]]

Caused by op 'magicpoint/eval_data_sharding/unstack_3', defined at:
File "experiment.py", line 151, in
args.func(config, output_dir, args)
File "experiment.py", line 89, in _cli_train
train(config, config['train_iter'], output_dir)
File "experiment.py", line 21, in train
with _init_graph(config) as net:
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/contextlib.py", line 82, in enter
return next(self.gen)
File "experiment.py", line 74, in _init_graph
n_gpus=n_gpus, **config['model'])
File "/home/ubuntu/Downloads/SuperPoint_Test2/SuperPoint-master/superpoint/models/base_model.py", line 122, in init
self._build_graph()
File "/home/ubuntu/Downloads/SuperPoint_Test2/SuperPoint-master/superpoint/models/base_model.py", line 265, in _build_graph
self._eval_graph(data)
File "/home/ubuntu/Downloads/SuperPoint_Test2/SuperPoint-master/superpoint/models/base_model.py", line 216, in _eval_graph
tower_metrics = self._gpu_tower(data, Mode.EVAL, self.config['eval_batch_size'])
File "/home/ubuntu/Downloads/SuperPoint_Test2/SuperPoint-master/superpoint/models/base_model.py", line 142, in _gpu_tower
shards = self._unstack_nested_dict(data, batch_size*self.n_gpus)
File "/home/ubuntu/Downloads/SuperPoint_Test2/SuperPoint-master/superpoint/models/base_model.py", line 126, in _unstack_nested_dict
else tf.unstack(v, num=num, axis=0) for k, v in d.items()}
File "/home/ubuntu/Downloads/SuperPoint_Test2/SuperPoint-master/superpoint/models/base_model.py", line 126, in
else tf.unstack(v, num=num, axis=0) for k, v in d.items()}
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1017, in unstack
return gen_array_ops.unpack(value, num=num, axis=axis, name=name)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 9138, in unpack
"Unpack", value=value, num=num, axis=axis, name=name)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/home/ubuntu/anaconda3/envs/SP_test/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Input shape axis 0 must equal 200, got shape [100,120,160]
[[Node: magicpoint/eval_data_sharding/unstack_3 = UnpackT=DT_INT32, axis=0, num=200, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: magicpoint/eval_tower2/map/while/box_nms/non_max_suppression/NonMaxSuppressionV3/_1553 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:2", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_1376_...pressionV3", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:2"]]

Does the code supports more than two GPUs?
Thank you.

Robust Descriptors

I read a few closed issues (#22) that you mentioned that you still hadn't gotten the descriptors to be very robust and therefore you can't really match features between two different images. I'm curious if you, or anyone, has since gotten the descriptors to work robustly? I think this is the best part of the original SuperPoint implementation by MagicLeap. From my experience, descriptor matching across images is super robust on their implementation.

If you, or anyone else has gotten it to work, could you please explain your methods a little bit so that I can replicate them?

Thanks!!!

Ground Truth of Synthetic Ellipses

Hi,
In file synthetic_dataset.py line # 331 why the function is returning np.empty((0, 2), dtype=np.int), should it not return the centers of ellipses?

train problem

I'm tring to train the Synthetic Shapes,but I GET THESE error;
can u tell me how to correct it,thanks!!!!!!!

INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
[10/30/2018 20:33:00 INFO] Scale of 0 disables regularizer.
2018-10-30 20:33:00.687044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN Xp, pci bus id: 0000:05:00.0, compute capability: 6.1)
2018-10-30 20:33:00.687071: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: TITAN Xp, pci bus id: 0000:06:00.0, compute capability: 6.1)
2018-10-30 20:33:00.687076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: TITAN Xp, pci bus id: 0000:09:00.0, compute capability: 6.1)
INFO:tensorflow:Start training
[10/30/2018 20:33:05 INFO] Start training
2018-10-30 20:33:06.198379: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: assertion failed: [None of the conditions evaluated as True. Conditions: (photometric_augmentation/while/Equal:0, photometric_augmentation/while/Equal_1:0, photometric_augmentation/while/Equal_2:0, photometric_augmentation/while/Equal_3:0, photometric_augmentation/while/Equal_4:0, photometric_augmentation/while/Equal_5:0), Values:] [0 0 0 1 0 0]
[[Node: photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert = Assert[T=[DT_STRING, DT_BOOL], summarize=6](photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/Switch, photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/data_0, photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/Switch_1)]]
Traceback (most recent call last):
File "/home/omnisky/anaconda3/envs/test_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/home/omnisky/anaconda3/envs/test_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/home/omnisky/anaconda3/envs/test_py3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [None of the conditions evaluated as True. Conditions: (photometric_augmentation/while/Equal:0, photometric_augmentation/while/Equal_1:0, photometric_augmentation/while/Equal_2:0, photometric_augmentation/while/Equal_3:0, photometric_augmentation/while/Equal_4:0, photometric_augmentation/while/Equal_5:0), Values:] [0 0 0 1 0 0]
[[Node: photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert = Assert[T=[DT_STRING, DT_BOOL], summarize=6](photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/Switch, photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/data_0, photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/Switch_1)]]
[[Node: magicpoint/IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?,?,1], [?,?,?], [?,?,2], [?,?,?]], output_types=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: magicpoint/train_data_sharding/stack_3/_9 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_77_magicpoint/train_data_sharding/stack_3", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/omnisky/0_2018_LIANG/2_DEEPLEARNING/1_DEPTH/learn-superpoint/SuperPoint-master/superpoint/experiment.py", line 148, in
args.func(config, output_dir, args)
File "/home/omnisky/0_2018_LIANG/2_DEEPLEARNING/1_DEPTH/learn-superpoint/SuperPoint-master/superpoint/experiment.py", line 86, in _cli_train
train(config, config['train_iter'], output_dir)
File "/home/omnisky/0_2018_LIANG/2_DEEPLEARNING/1_DEPTH/learn-superpoint/SuperPoint-master/superpoint/experiment.py", line 27, in train
keep_checkpoints=config.get('keep_checkpoints', 1))
File "/home/omnisky/0_2018_LIANG/2_DEEPLEARNING/1_DEPTH/learn-superpoint/SuperPoint-master/superpoint/superpoint/models/base_model.py", line 313, in train
options=options, run_metadata=run_metadata)
File "/home/omnisky/anaconda3/envs/test_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/omnisky/anaconda3/envs/test_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/omnisky/anaconda3/envs/test_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/omnisky/anaconda3/envs/test_py3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [None of the conditions evaluated as True. Conditions: (photometric_augmentation/while/Equal:0, photometric_augmentation/while/Equal_1:0, photometric_augmentation/while/Equal_2:0, photometric_augmentation/while/Equal_3:0, photometric_augmentation/while/Equal_4:0, photometric_augmentation/while/Equal_5:0), Values:] [0 0 0 1 0 0]
[[Node: photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert = Assert[T=[DT_STRING, DT_BOOL], summarize=6](photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/Switch, photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/data_0, photometric_augmentation/while/case/If_0/Assert_1/AssertGuard/Assert/Switch_1)]]
[[Node: magicpoint/IteratorGetNext = IteratorGetNextoutput_shapes=[[?,?,?,1], [?,?,?], [?,?,2], [?,?,?]], output_types=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[Node: magicpoint/train_data_sharding/stack_3/_9 = _HostRecvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_77_magicpoint/train_data_sharding/stack_3", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]