leedoyup / anogan-tf

Unofficial Tensorflow Implementation of AnoGAN (Anomaly GAN)

Hello 😄

I am Doyup Lee. Now, I am an AI researcher at Runway, New York.
I definetly believe that machine learning can make human life much happier.
Of course, I love to learn new technologies.

Research Interests

• Large-Scale Generative Models
• Multi-modal and unified representation learning
• Training DNNs with limited labels
• Robustness of deep learning and Anomaly Detection
• Technology Innovation

Contact

• e-mail: [email protected]
• google scholar: https://scholar.google.co.kr/citations?user=doyuplee&user=5rAj44kAAAAJ
• linked-in: https://www.linkedin.com/in/doyuplee/
• facebook: https://www.facebook.com/doyup2

anogan-tf's Issues

Error about assertion while running code on celebA dataset

Problem while Training 2nd time on custom data

Hello,

I successfully trained yesterday on a custom dataset, and when I tried to retrain my model, it throws the following error:

[] Reading checkpoints...
[] Success to read DCGAN.model-64502
[*] Load SUCCESS
2019-01-30 09:23:31.722534: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2019-01-30 09:23:31.722578: E tensorflow/stream_executor/cuda/cuda_dnn.cc:381] Possibly insufficient driver version: 384.130.0
2019-01-30 09:23:31.722589: W ./tensorflow/stream_executor/stream.h:2093] attempting to perform DNN operation using StreamExecutor without DNN support
2019-01-30 09:23:31.722603: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2019-01-30 09:23:31.722641: E tensorflow/stream_executor/cuda/cuda_dnn.cc:381] Possibly insufficient driver version: 384.130.0
2019-01-30 09:23:31.722650: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2019-01-30 09:23:31.722665: E tensorflow/stream_executor/cuda/cuda_dnn.cc:381] Possibly insufficient driver version: 384.130.0
Traceback (most recent call last):
File "main.py", line 124, in
tf.app.run()
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "main.py", line 106, in main
dcgan.train(FLAGS)
File "/home/lipika/AnoGAN/model.py", line 239, in train
_, summary_str = self.sess.run([d_optim, self.d_sum], feed_dict = d_feed_dict)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: cuDNN launch failure : input shape ([4,512,31,32])
[[node generator/g_bn0/FusedBatchNorm (defined at /home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py:368) = FusedBatchNorm[T=DT_FLOAT, data_format="NCHW", epsilon=1.001e-05, is_training=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](generator/g_bn0/FusedBatchNorm-0-TransposeNHWCToNCHW-LayoutOptimizer, generator/g_bn0/gamma/read, generator/g_bn0/beta/read, discriminator/d_bn1/Const, discriminator/d_bn1/Const)]]
[[{{node add/_109}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_595_add", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op u'generator/g_bn0/FusedBatchNorm', defined at:
File "main.py", line 124, in
tf.app.run()
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "main.py", line 101, in main
test_dir = FLAGS.test_dir)
File "/home/lipika/AnoGAN/model.py", line 75, in init
self.build_model()
File "/home/lipika/AnoGAN/model.py", line 112, in build_model
self.G = self.generator(self.z, self.y)
File "/home/lipika/AnoGAN/model.py", line 346, in generator
h0 = tf.nn.relu(self.g_bn0(self.h0))
File "/home/lipika/AnoGAN/ops.py", line 34, in call
scope=self.name)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 596, in batch_norm
scope=scope)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 382, in _fused_batch_norm
is_training, _fused_batch_norm_training, _fused_batch_norm_inference)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/utils.py", line 214, in smart_cond
return static_cond(pred_value, fn1, fn2)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/utils.py", line 192, in static_cond
return fn1()
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 368, in _fused_batch_norm_training
inputs, gamma, beta, epsilon=epsilon, data_format=data_format)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/nn_impl.py", line 909, in fused_batch_norm
name=name)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 3466, in _fused_batch_norm
is_training=is_training, name=name)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

InternalError (see above for traceback): cuDNN launch failure : input shape ([4,512,31,32])
[[node generator/g_bn0/FusedBatchNorm (defined at /home/lipika/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py:368) = FusedBatchNorm[T=DT_FLOAT, data_format="NCHW", epsilon=1.001e-05, is_training=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](generator/g_bn0/FusedBatchNorm-0-TransposeNHWCToNCHW-LayoutOptimizer, generator/g_bn0/gamma/read, generator/g_bn0/beta/read, discriminator/d_bn1/Const, discriminator/d_bn1/Const)]]
[[{{node add/_109}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_595_add", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Could you please assist me with this?

Thank you :)

datasetsize

my data size is 1500*1200,can I set the input_height and input_width with 1500 and 1200?and is there some requirement about the input_height,input_width,and output_height,putput_width?

Query regarding loading weight file during testing

Hi,
Thank you very much for providing the code. Since I am new to tensorflow, I was wondering if anyone could guide me on where is my trained weight file being loaded during testing?

Thanks

TabError: inconsistent use of tabs and spaces in indentation

(tensorflow) cardiac@cardiac-All-Series:~/AnoGAN$ python main.py --dataset=celebA input_height=108 --train
File "main.py", line 115
test_input = np.expand_dims(dcgan.test_data[idx],axis=0)
^
TabError: inconsistent use of tabs and spaces in indentation

After running main.py, I simply get the "TabError". Can you point me to how to solve it is?

Where is the most similar image used construct the Residual Loss, iterative choose process ?

It seems like, the Residual Loss construct is used test data in one dictionary, so can you describe the criterion code of choose most similar image in the train data ?

everyone,I want to know the test_result of yours

is well the test result of your own dataset ,and the generated picture is similar with the test picture?

download celebA error

[python download.py celebA] cannot work well. help me

This is amazing! Thank you!

Hi, i was wondering what 'h1, h2, h3, h4' is.
Are those the hidden layers? If not, are there any hidden layers in this code?
Thank you!

About a Anomaly Detection error

Hi ! LeeDoYup
First, Thank you for your share!

I use your code with my dataset to train DCGAN normally.After having trained DCGAN model, when I test with my data Using:
$ python main.py --dataset DATASET_NAME --input_height=108 --crop --anomaly_test

There always be a error:

Traceback (most recent call last):
File "main.py", line 124, in
tf.app.run()
File "/home/motorzhu/anaconda3/envs/py2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "main.py", line 101, in main
test_dir = FLAGS.test_dir)
File "/home/motorzhu/Download/AnoGAN-master/model.py", line 67, in init
imreadImg = imread(self.data[0])
IndexError: list index out of range

I do put my test data in "test_data",but it shows the model can‘t read the image,can you give me the right anomaly test command ($ python main.py --dataset DATASET_NAME --input_height=108 --crop --anomaly_test) please? Thank you !

anomaly_test

can I use a lot of picture doing the anomaly_test rather than one picture

anomaly test(Looking for your reply)

请问数据集的图片大小是可以自己调节的吗？

我一开始把input_height设置成256，图片256结果报错
ValueError: Trying to share variable discriminator/d_h4_lin/Matrix, but specified shape (8192, 1) and found shape (131072, 1)

后来，使用默认108，图片也用108*108的，但是还是报错
ValueError: Trying to share variable discriminator/d_h4_lin/Matrix, but specified shape (8192, 1) and found shape (25088, 1).

请问怎么样规格的图片才可以让程序接纳？😆

questions about the input_height and input_width

last time,I use my data with the size of 15441284,and I resized it with 178218 as the celeba dataset,but the result is not good.The generated picture is not clear,so I want to ask how to choose the size of data

About testing slowing down!!

Hey Lee,

I implemented your AnoGAN, but training z-dim noise slowing down during testing in each iteration, do you have any solution to solve the issue? I guess tensorflow graph memory isn't release?

If you can fix the issue, i will thank you very much!!!

Best Regards.

William

How files in checkpoint are created ?

I am working on a problem that my dataset is divided in subsets of 10. So what i need to do is train each subset of data with few epoches then stop training and again start the training process by replacing the previous subset of data with new one. Now the problem is that in the final or last model file in checkpoint does not contain any information of training data of previous subsets of dataset i trained it on but contain information on only the subset of dataset i trained at the end ........ So my question is ... Isn't model file should contain all info of all subsets of dataset i trained on?

The problem is related to additional training on already trained data....
Please help @LeeDoYup

ValueError: Trying to share variable discriminator/d_h4_lin/Matrix, but specified shape (8192, 1) and found shape (25088, 1).

Traceback (most recent call last):
File "main.py", line 124, in
tf.app.run()
File "/home/li/tensorflow/lib/python3.4/site-packages/tensorflow/python/platform/app.py", line 48, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "main.py", line 101, in main
test_dir = FLAGS.test_dir)
File "/home/li/gan/AnoGAN-master/model.py", line 75, in init
self.build_model()
File "/home/li/gan/AnoGAN-master/model.py", line 116, in build_model
self.D, self.D_logits = self.discriminator(self.G, self.y, reuse=True)
File "/home/li/gan/AnoGAN-master/model.py", line 283, in discriminator
h4 = linear(tf.reshape(h3, [batch_size, -1]), 1, 'd_h4_lin')
File "/home/li/gan/AnoGAN-master/ops.py", line 80, in linear
tf.random_normal_initializer(stddev=stddev))
File "/home/li/tensorflow/lib/python3.4/site-packages/tensorflow/python/ops/variable_scope.py", line 1065, in get_variable
use_resource=use_resource, custom_getter=custom_getter)
File "/home/li/tensorflow/lib/python3.4/site-packages/tensorflow/python/ops/variable_scope.py", line 962, in get_variable
use_resource=use_resource, custom_getter=custom_getter)
File "/home/li/tensorflow/lib/python3.4/site-packages/tensorflow/python/ops/variable_scope.py", line 367, in get_variable
validate_shape=validate_shape, use_resource=use_resource)
File "/home/li/tensorflow/lib/python3.4/site-packages/tensorflow/python/ops/variable_scope.py", line 352, in _true_getter
use_resource=use_resource)
File "/home/li/tensorflow/lib/python3.4/site-packages/tensorflow/python/ops/variable_scope.py", line 669, in _get_single_variable
found_var.get_shape()))
ValueError: Trying to share variable discriminator/d_h4_lin/Matrix, but specified shape (8192, 1) and found shape (25088, 1).

the difference between test image and generated image

Is there a big difference between the generated image and the test one?and what is your test result?is it very similar between the test image and the generated image?

AssertionError

I used the celebA and met the trouble：

Epoch: [ 0] [ 96/6331] time: 20.7979, d_loss: 2.24110365, g_loss: 0.32057661
Epoch: [ 0] [ 97/6331] time: 20.9898, d_loss: 1.89617133, g_loss: 0.46526995
Epoch: [ 0] [ 98/6331] time: 21.1837, d_loss: 1.54140878, g_loss: 0.72463155
Traceback (most recent call last):
File "main.py", line 124, in
tf.app.run()
File "/home/wangyun/anaconda3/envs/AnoGAN/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "main.py", line 106, in main
dcgan.train(FLAGS)
File "/home/wangyun/anaconda3/envs/AnoGAN/GAN/model.py", line 265, in train
save_images(samples, image_manifold_size(samples.shape[0]),
File "/home/wangyun/anaconda3/envs/AnoGAN/GAN/utils.py", line 197, in image_manifold_size
assert manifold_h * manifold_w == num_images
AssertionError

Can you direct me？Thank you.

Detect Anomaly on my own dataset

I want to train the GAN on my own dataset, after training for about 1000 epochs, the score of abnormal data don't have a gap with normal data. Can you give me some comments about the training tricks?

MNIST

Higher Resolution Images

Hi,
I'm trying to use my own dataset, where I can't compromise much with the resolution of the image. Original resolution of the image being 800 x 800, I have tried reducing it up to 256 x 256, this still gives "ResourceExhaustedError".
I'm using google colab for training the model which gives 16GB Tesla K80 GPU.
Can you suggest anything for this?

downlaod

Multi-GPU training

Hello,

how could we train on multiple GPUs? On my system it seems only one GPU is loaded.

The configuration of my system is:
Ubuntu 16.04 LTS
2 x GeForce GTX 1080 8GB

Thank you in advance!

additional training ..

Thank you for your AnoGAN.

I have two questions.
First, It goes out because of a lack of capacity while studying.
I use my data with the size of 1024, batch size of 1

Second, I want to additional training.
What should I say on the command ?

I wonder that 'ano_z' variable should be ...

I think that 'ano_z' variable should be resampled from latent space distribution randomly every a test data. Am I right?

At train AnoGAN with MNIST data, Does the train set and test set all contain only normal data?

Hi, @LeeDoYup

In the case of Anomaly Gan, I knew that it trained with only normal datasets.

At train AnoGAN with MNIST data, Does the train set and test set all contain only normal data?

Or are both normal + abnormal divided into folders?
I wonder.

form @bemoregt.

Interpreting the results

Hi,

The generated and the test images are not same ( this was tested with test_epoch=100 so I got 3 generated images, first one at 0th epoch, 50th epoch and 99th epoch). Trying on grey scale image set but making the required changes to the code.

The score are as follows:
Epoch: [00], anomaly score: 50176.87890625, res loss: 48897.88281250
Epoch: [50], anomaly score: 38163.79296875, res loss: 35445.21093750
Epoch: [99], anomaly score: 27550.78906250, res loss: 23283.65039062

I am having trouble to know what these scores mean. Using these numbers how should I go about identifying if the test image as an anomaly or not ? What could be the good threshold value etc etc. Please assist.

For training I used about 1000 normal images with 1000 epochs.

thanks,
Pradeep

every pixel anomal_score

You said that the anomaly_score is the sum of the all pixels anomaly_score,if I want to see the every pixel's anomaly_score,what shoul I do ?
the code"self.res_loss = tf.reduce_mean(
tf.reduce_sum(tf.abs(tf.subtract(test_inputs, self.ano_G))))" is the whole pixel's res_loss ?it is 6464 and 3 channels,but for the dis_loss : "self.dis_loss = tf.reduce_mean(
tf.reduce_sum(tf.abs(tf.subtract(dis_f_z, dis_f_input))))",it is 66 and 512 channels?,what should I do if I want to know every pixel?

the result of my dataset

When I use my own dataset ,the generated picture are so blurred,and seems like the same color.I do not know what is the matter

License

In your acknowledgements, you write: "Thanks for @carpedm20 's implementation of DCGAN. I implemented AnoGAN based on his implementation."

DCGAN is published under MIT license. https://github.com/carpedm20/DCGAN-tensorflow/blob/master/LICENSE

Please update the license of your repository because otherwise it is unclear for others to license their product which may be based on your code.

about the gray dataset

I try the gray dataset to train,but in the test stage,there is no test_result folder ,and the grayscale is True，it has no result images

About t-SNE embedding

Recently I am learning about embedding, could I ask about the implementations of t-SNE embedding on the feature representation of the last convolution layer of the discriminator in your research.Thanks a lot!

Anomaly detection slowing down

Hi Lee,

Thanks for sharing your AnoGAN implementation.
I implemented your version and noticed that when looping on anomaly detection the process goes slower an slower. Ater 200 iterrations, the computation can takes more than on minute !
Do you have an idea on what can cause that behaviour ?

Testing trained anomaly detector: Error: None values not supported

To repeat the issue:
1) Download data:
python download.py mnist
2) Train a model
python main.py --dataset mnist --input_height=28 --output_height=28 --train
3) Prepare test data
Downloaded sample images and put them to test_data folder
4) Run anomaly detection
python main.py --dataset mnist --batch_size=64 --input_height=28 --output_height=28 --crop --anomaly_test

Error:

  File "main.py", line 129, in <module>
    tf.app.run()
  File "/home/.conda/envs/AnoGan/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/.conda/envs/AnoGan/lib/python2.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/.conda/envs/AnoGan/lib/python2.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "main.py", line 117, in main
    dcgan.anomaly_detector()
  File "/home/NoveltyDetection/AnoGan/model.py", line 487, in anomaly_detector
    self.ano_G = self._sampler(self.ano_z, self.ano_y, batch_size=1)
  File "/home/NoveltyDetection/AnoGan/model.py", line 431, in _sampler
    yb = tf.reshape(y, [self.batch_size, 1, 1, self.y_dim])
  File "/home/.conda/envs/AnoGan/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7715, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/home/.conda/envs/AnoGan/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 545, in _apply_op_helper
    (input_name, err))
ValueError: Tried to convert 'tensor' to a tensor and failed. Error: None values not supported.

Cause:
https://github.com/LeeDoYup/AnoGAN/blob/4c650b3ab606a810c63cbb1f8289debbe25bb405/model.py#L478

https://github.com/LeeDoYup/AnoGAN/blob/4c650b3ab606a810c63cbb1f8289debbe25bb405/model.py#L428

Environment:
Python 2.7
Tensorflow 1.14.0

Experiment no. 1 trying to solve the issue:
self.ano_y = tf.placeholder(tf.float32, [self.test_batch_size, self.y_dim], name='y')
Output:
ValueError: Cannot reshape a tensor with 10 elements to shape [64,1,1,10] (640 elements) for 'generator_2/Reshape' (op: 'Reshape') with input shapes: [1,10], [4] and with input tensors computed as partial shapes: input[1] = [64,1,1,10].

similar results also with [1, 1, 1, self.y_dim] or [1, 1, self.y_dim]

anomaly score

When I finished the anomaly test stage with the anomal score,and how can I know the anomaly picture

errors in the code

hey, i recently cloned this repo, there are a few errors in the code, i couldn't train my dataset.

테스트를 해보니 에러는 안나는데 test_data 폴더에 test_result 폴더가 안생깁니다.

Hi, @LeeDoYup

내 이미지 데이타셋으로 학습까지 잘 마쳤는데
테스트를 해보니 에러는 안나는데 test_data 폴더에 test_result 폴더가 안생깁니다.

왜 이런걸까요?

내 학습명령:
$ python main.py --dataset ELX --train --input_height=128 --crop

내 테스트명령:
$ python main.py –-dataset ELX –-input_height=64 --crop –-anomaly_test

참고로 내 데이타는 265x265 크기의 그레이스케일 이미지들 입니다.

미리, 감사드리며~
from @bemoregt.

Some Questions on Test stage

The first stage training is very effective, but when I want to test the accuracy, some issues turned out:

1. in the model.py anomaly_detector() self.ano_y = none. it will report an error in the def _sampler:
   yb = tf.reshape(y, [-1, 1, 1, self.y_dim]),because none cannot be reshaped.
   2.and when test, the batch_size is set to 1, the self.batch_size in _sampler should also be set to 1.
   3.After I modify these codes, and change the input from images into the same data with train, the samples are just like some noises, and I wonder why.

Thank you for your AnoGAN............

First of all, thank you for your ANOGAN.
I have a one problem.
I'm sure the learning went well, but there's an error.

Exception: [!] Train a model first, then run test mode

Do you know what the problem is?
Thank you .

attributeError: 'DCGAN' object has no attribute 'test_data_names'

anomal-area

errors = np.mean(np.squeeze(errors),axis=2)
the error is the anomal-score image,and this function of the code is that using the gray-level to label the anomal-area?

Code to highlight the anomaly detected areas

Could you please tell me how to highlight the anomaly detected regions (red color) for X-ray unseen images like you demonstrated in the readme ?

Thanks.

Expect argument to be a string or int, found

I am facing this problem.

$ python main.py --dataset cresco --train Traceback (most recent call last): File "main.py", line 20, in <module> flags.DEFINE_integer("train_size", np.inf, "The size of train images [np.inf]") File "/home/uzzal/miniconda2/envs/tf_gpu_py2/lib/python2.7/site-packages/tensorflow/python/platform/flags.py", line 58, in wrapper return original_function(*args, **kwargs) File "/home/uzzal/miniconda2/envs/tf_gpu_py2/lib/python2.7/site-packages/absl/flags/_defines.py", line 315, in DEFINE_integer DEFINE(parser, name, default, help, flag_values, serializer, **args) File "/home/uzzal/miniconda2/envs/tf_gpu_py2/lib/python2.7/site-packages/absl/flags/_defines.py", line 81, in DEFINE DEFINE_flag(_flag.Flag(parser, serializer, name, default, help, **args), File "/home/uzzal/miniconda2/envs/tf_gpu_py2/lib/python2.7/site-packages/absl/flags/_flag.py", line 107, in __init__ self._set_default(default) File "/home/uzzal/miniconda2/envs/tf_gpu_py2/lib/python2.7/site-packages/absl/flags/_flag.py", line 196, in _set_default self.default = self._parse(value) File "/home/uzzal/miniconda2/envs/tf_gpu_py2/lib/python2.7/site-packages/absl/flags/_flag.py", line 169, in _parse 'flag --%s=%s: %s' % (self.name, argument, e)) absl.flags._exceptions.IllegalFlagValueError: flag --train_size=inf: Expect argument to be a string or int, found <type 'float'>