david-gpu / deep-makeover Goto Github PK

Hello
I have such when i try to start the training:
python3 dm_main.py --run train
Traceback (most recent call last):
File "dm_main.py", line 170, in
tf.app.run()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "dm_main.py", line 160, in main
train_data = _get_train_data()
File "dm_main.py", line 93, in _get_train_data
source_images = dm_input.input_data(sess, 'train', selected)
File "/home/deep-makeover/dm_input.py", line 11, in input_data
assert mode == 'inference' or len(filenames) >= FLAGS.test_vectors
AssertionError

I don't understand where I can get this dataset?
After you have the required software above you will also need the Large-scale CelebFaces Attributes (CelebA) Dataset. The model expects the Align&Cropped Images version. Extract all images to a subfolder named dataset. I.e. deep-makeover/dataset/lotsoffiles.jpg.

Hey! I installed tenseflow by using anaconda, by instructions. but I have some problems please help!
When I run in terminal => python3 dm_main.py --run train
it give back this error:
Traceback (most recent call last):
File "dm_main.py", line 18, in
import dm_model
File "/Users/oteldanagul/Downloads/deep-makeover-master/dm_model.py", line 5, in
import dm_arch
File "/Users/oteldanagul/Downloads/deep-makeover-master/dm_arch.py", line 10, in
_glbl_is_training = tf.Variable(initial_value=True, trainable=False, name='glbl_is_training')
AttributeError: module 'tensorflow' has no attribute 'Variable'

Hey thanks for the code – the results look great! Can't wait to try it myself :-)

Managed to install tensorflow with bazel from source - with CUDA 8 and cudnn 5 support (on osx 10.12 with nvidia gtx 780 with 6gb VRAM).

Can run tensorflow test files without errors. Cuda gets detected and works.

The CelebA dataset + txt files are in the dataset directory and seem to get loaded successfully.

But after starting dm_main.py these errors appear:

$ python3 dm_main.py --run train
   69714 source images selected
   78610 target images selected

Generator input (feature) size is 100 x 80 x 3 = 24000
Generator has 0.59M parameters

Discriminator input (feature) size is 100 x 80 x 3 = 24000
Discriminator has 0.84M parameters

Building testing model...
Done.

Model training...
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1022, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1004, in _run_fn
    status, run_metadata)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[16,100,80,48]
	 [[Node: BiasAdd_16 = BiasAdd[T=DT_FLOAT, data_format="NHWC", _device="/job:localhost/replica:0/task:0/gpu:0"](Conv2D_16, GENE/L052/bias/read)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dm_main.py", line 169, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "dm_main.py", line 160, in main
    dm_train.train_model(train_data)
  File "/Users/david/github/image-manipulation/deep-makeover/dm_train.py", line 99, in train_model
    td.sess.run(minimize_ops)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 767, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 965, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[16,100,80,48]
	 [[Node: BiasAdd_16 = BiasAdd[T=DT_FLOAT, data_format="NHWC", _device="/job:localhost/replica:0/task:0/gpu:0"](Conv2D_16, GENE/L052/bias/read)]]

Caused by op 'BiasAdd_16', defined at:
  File "dm_main.py", line 169, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "dm_main.py", line 159, in main
    train_data = _get_train_data()
  File "dm_main.py", line 106, in _get_train_data
    train_model  = dm_model.create_model(sess, source_images, target_images, annealing, verbose=True)
  File "/Users/david/github/image-manipulation/deep-makeover/dm_model.py", line 174, in create_model
    gene          = _generator_model(sess, source_images)
  File "/Users/david/github/image-manipulation/deep-makeover/dm_model.py", line 103, in _generator_model
    _residual_block(model, nunits, mapsize)
  File "/Users/david/github/image-manipulation/deep-makeover/dm_model.py", line 65, in _residual_block
    model.add_conv2d(num_units, mapsize=1, stride=1)
  File "/Users/david/github/image-manipulation/deep-makeover/dm_arch.py", line 235, in add_conv2d
    out    = tf.nn.bias_add(out, bias)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 1316, in bias_add
    return gen_nn_ops._bias_add(value, bias, data_format=data_format, name=name)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 281, in _bias_add
    data_format=data_format, name=name)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2402, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1264, in __init__
    self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[16,100,80,48]
	 [[Node: BiasAdd_16 = BiasAdd[T=DT_FLOAT, data_format="NHWC", _device="/job:localhost/replica:0/task:0/gpu:0"](Conv2D_16, GENE/L052/bias/read)]]

Does the "ResourceExhaustedError" mean the gpu is not available for processing? Or that i don´t have enough free VRAM?

How could i reduce the memory required – what would i need to change in your files?

Progress[100%], ETA[ 0m], Batch [145460], gene[0.203], disc[0.552] real[0.218] fake[0.334]
Traceback (most recent call last):
File "dm_main.py", line 169, in
tf.app.run()
File "/usr/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "dm_main.py", line 160, in main
dm_train.train_model(train_data)
File "/mnt/disk02/projects/deep-makeover_ftm/dm_train.py", line 132, in train_model
_save_checkpoint(td, batch)
File "/mnt/disk02/projects/deep-makeover_ftm/dm_train.py", line 68, in _save_checkpoint
saver.save(td.sess, newname)
File "/usr/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1386, in save
self.export_meta_graph(meta_graph_filename)
File "/usr/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1414, in export_meta_graph
graph_def=ops.get_default_graph().as_graph_def(add_shapes=True),
File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2257, in as_graph_def
result, _ = self._as_graph_def(from_version, add_shapes)
File "/usr/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2220, in _as_graph_def
raise ValueError("GraphDef cannot be larger than 2GB.")
ValueError: GraphDef cannot be larger than 2GB.

During the making training by use of => python3 dm_main.py --run train
it is generated only three bathes, but result not so pretty good. So I started training again, but results are similar to previous. @david-gpu How much time I should run this command? help me

@david-gpu
I download Large-scale CelebFaces Attributes (CelebA) Dataset from here = > http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
and I dropped all images to dataset and created list_attr_celeba.txt file.
But I have that problem when running command "python3 dm_main.py --run train" =>

Traceback (most recent call last):
File "dm_main.py", line 169, in
tf.app.run()
File "/Users/oteldanagul/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "dm_main.py", line 159, in main
train_data = _get_train_data()
File "dm_main.py", line 93, in _get_train_data
source_images = dm_input.input_data(sess, 'train', selected)
File "/Users/oteldanagul/Downloads/deep-makeover-master/dm_input.py", line 11, in input_data
assert mode == 'inference' or len(filenames) >= FLAGS.test_vectors
AssertionError

Hello,

I am interested in your project, are there any papers that I can refer to?

Thanks a lot :)

Hey i finally got your code running – the results are really amazing! Will show them to my boss on monday :-)

I have a few questions/requests:

Longer training – better results?
What would i need to change in the files to let the training run longer, to get better results? It was also interesting that my gtx 780 (like your 1080) only took 2 hours to complete the training. I thought an older gpu would take a lot more time.

Does a longer training get you clearer pictures (less artifacts)?

Higher resolution images
Would it be possible to generate images with 2x or larger resolution? I know you would also need high res training images – but i think there is another version of the celeba dataset with bigger pictures (i found two archives with ~12gb, but haven't downloaded them yet).

What would i need to modify in your files?

Pictures with full head, not only cropped below hair
Would this also be possible? It would be nice to get a bigger frame of the person. Or does your code only work for facial features?

Input arguments for image generation
Could you please add a few lines to your readme on what options you can use to generate the output?

Hope i'm not asking too much – but it would be great if you could generate images that could be printed. At least on a small scale. I know i could also use a neural net project to increase the resolution (your srez code) – but it's hard to get much more detail out of a 80x100px file.

Thanks again for your work! I really enjoy testing your projects, the srez one also was fun to try out!

Results not repeatable?

I installed all the necessary files and started it with the default values. Training worked fine I assume (trained for 3 hours), final training batch looks like this:

But when I open an image for inference, in this case image number 201739 from the celeb a dataset, the result of the inference (which should be male to female) is:

Can you tell me what is going wrong?

How to get the key in dm_input.py??

I'm a newbie in deep learning and found your code very pretty to start dive )))
I found some problem to run next command:
python3.5 dm_main.py --run train

You can check my changes in this place:
https://github.com/chapayGhub/deep-makeover/tree/feature/test

briefly:
in several places

• I changed labels= to targets= as sample dm_model.py

tf.nn.sigmoid_cross_entropy_with_logits(logits=disc_fake_output, labels=tf.ones_like(disc_fake_output))  
to      
tf.nn.sigmoid_cross_entropy_with_logits(logits=disc_fake_output, targets=tf.ones_like(disc_fake_output))

• I changed order of input parameters for tf.concat function as sample dm_train.py

image    = tf.concat([feature, clipped], 2)
image    = tf.concat(2, [feature, clipped])

Yooo hooo
as result I can started training now!!!

python3.5 dm_main.py --run train:

Connected to pydev debugger (build 163.10154.50)
   69714 source images selected
   78610 target images selected

Generator input (feature) size is 100 x 80 x 3 = 24000
Generator has 0.59M parameters

Discriminator input (feature) size is 100 x 80 x 3 = 24000
Discriminator has 0.84M parameters

Building testing model...
Done.

Model training...
  Progress[  0%], ETA[ 119m], Batch [    0], gene[0.105], disc[1.384] real[0.716] fake[0.669]
    Saved train/batch000000_out.png
  Progress[  1%], ETA[ 117m], Batch [   10], gene[0.083], disc[1.315] real[0.670] fake[0.645]
  Progress[  2%], ETA[ 116m], Batch [   20], gene[0.100], disc[1.304] real[0.705] fake[0.599]
  ..................................................................
  Progress[ 49%], ETA[  60m], Batch [  600], gene[0.096], disc[1.101] real[0.575] fake[0.525]
    Saved train/batch000600_out.png
  Progress[ 50%], ETA[  59m], Batch [  610], gene[0.128], disc[1.274] real[0.603] fake[0.672]
  Progress[ 50%], ETA[  59m], Batch [  620], gene[0.104], disc[1.569] real[0.945] fake[0.624]
  ..................................................................
  Progress[ 95%], ETA[   5m], Batch [  660], gene[0.099], disc[1.356] real[0.754] fake[0.602]
  Progress[225%], ETA[-151m], Batch [  670], gene[0.079], disc[1.350] real[0.572] fake[0.778]
    Checkpoint saved
Finished training!

And we can run:
python3 dm_main.py --run inference --infile ../test/arnold.schwarzenegger.jpg --outfile ../test/arnold.schwarzenegger.out.jpg

Don't worry Arnie isn't my boss ))). Only for testing!

And got result:
arnold.schwarzenegger.jpg:

arnold.schwarzenegger.out.jpg:

So I have questions:

• Are my code changes is right?
• Is it good result? I wanted to make Arnie more pretty, but made him like Terminator from broken old TV ))
• At first I want to achieve result that you have in example_male_to_female.jpg . What should i do?

PS
Thank you
Your code very readable and moduled, it very help me!