python3 finetuning errors about ctrl

[xurongqiang]

python2 finetuning seems good, but Resource exhausted: OOM(P100)
19-09-27 14:29:48.886518: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba12398600 next 2020 of size 5120
2019-09-27 14:29:48.886523: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba12399a00 next 2021 of size 5120
2019-09-27 14:29:48.886527: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1239ae00 next 2022 of size 6553600
2019-09-27 14:29:48.886538: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba129dae00 next 2023 of size 6553600
2019-09-27 14:29:48.886543: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1301ae00 next 2024 of size 5120
2019-09-27 14:29:48.886548: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1301c200 next 2025 of size 41943040
2019-09-27 14:29:48.886552: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1581c200 next 2026 of size 6553600
2019-09-27 14:29:48.886557: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba15e5c200 next 2027 of size 5120
2019-09-27 14:29:48.886561: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba15e5d600 next 2028 of size 6553600
2019-09-27 14:29:48.886566: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1649d600 next 2029 of size 5120
2019-09-27 14:29:48.886570: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1649ea00 next 2030 of size 5120
2019-09-27 14:29:48.886575: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1649fe00 next 2031 of size 5120
2019-09-27 14:29:48.886579: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba164a1200 next 2032 of size 5120
2019-09-27 14:29:48.886584: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba164a2600 next 2033 of size 5120
2019-09-27 14:29:48.886588: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba164a3a00 next 2034 of size 6553600
2019-09-27 14:29:48.886593: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba16ae3a00 next 2035 of size 6553600
2019-09-27 14:29:48.886597: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba17123a00 next 2036 of size 41943040
2019-09-27 14:29:48.886602: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba19923a00 next 2037 of size 5120
2019-09-27 14:29:48.886606: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba19924e00 next 2038 of size 6553600
2019-09-27 14:29:48.886611: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba19f64e00 next 2039 of size 41943040
2019-09-27 14:29:48.886615: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1c764e00 next 2040 of size 41943040
2019-09-27 14:29:48.886620: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1ef64e00 next 2041 of size 5120
2019-09-27 14:29:48.886624: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1ef66200 next 2042 of size 5120
2019-09-27 14:29:48.886629: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1ef67600 next 2043 of size 5120

[keskarnitish]

It's a bit difficult to debug since I don't have the whole log file but from what you've posted, it seems like you didn't (re-?)patch keras.py to have use_tpu=False?