skyflynil / stylegan2 Goto Github PK

I have the following errors when trying to train 3x2 (768x512) model:

assert min_h > 2 and min_w>2 and res_log2>=1

around line 542 and 786 in /training/networks_stylegan2.py

Is there a reason for this limitation? I changed it to min_w >= 2 and will update this with my results.

Hi everyone, I've been tinkering with stylegan2 for the past week or 2 and have stumbled across this page to be able to train portrait versions of pictures.

I have thus stumbled across an issue when trying to train the data I have prepared.

python run_training.py --num-gpus=1 --use-raw=false --data-dir=D:\Training_Data\FP_Project\Resize_Images --config=config-f --dataset=D:\Training_Data\FP_Project\Dest_Images --mirror-augment=true --metric=none --total-kimg=20000 --min-h=10 --min-w=6 --res-log2=7 --result-dir='D:\Training_Data\FP_Project\Results'

Upon running the above code I encountered the error:
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'D:\\Training_Data\\FP_Project\\Results\\00000-stylegan2-D:'

I have tried various fixes on stackoverflow such as encapsulating the --result-dir in double quotes("") or single quotes('') and even adding double back slashes(\) in between the file directories. But to no avail.
I have even tried placing a 'r' in front of the file path to try designating the filepath as a raw string, but it resulted in this error:
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'rD:'

I went as far as to edit the default value of '--result-dir' and running 'run_training.py' without --result-dir as part of the argument. All attempts resulted in the same error.

If anyone could provide any solutions as to how I should go about fixing this issue it would be very much appreciated.

Thanks in advance!!!

I have posted the question on stack overflow below is the link to it:
Stylegan2 model with flask API is generating weird results after first request

Please help me with this issue, I have no idea how to fix it.

when I run:
python run_generator.py generate-images --network=weights/stylegan2-car-config-f.pkl --seeds=6000-6025 --truncation-psi=0.5
error occurd:
Traceback (most recent call last):
File "run_generator.py", line 168, in
main()
File "run_generator.py", line 163, in main
dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/submission/submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/submission/internal/local.py", line 22, in submit
return run_wrapper(submit_config)
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/submission/submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "/home/fut/Downloads/stylegan2-NVlabs/run_generator.py", line 21, in generate_images
_G, _D, Gs = pretrained_networks.load_networks(network_pkl)
File "/home/fut/Downloads/stylegan2-NVlabs/pretrained_networks.py", line 76, in load_networks
G, D, Gs = pickle.load(stream, encoding='latin1')
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/tflib/network.py", line 297, in setstate
self._init_graph()
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/tflib/network.py", line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File "", line 491, in G_synthesis_stylegan2
File "", line 455, in layer
File "", line 99, in modulated_conv2d_layer
File "", line 68, in apply_bias_act
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/tflib/ops/fused_bias_act.py", line 68, in fused_bias_act
return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/tflib/ops/fused_bias_act.py", line 122, in _fused_bias_act_cuda
cuda_kernel = _get_plugin().fused_bias_act
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/tflib/ops/fused_bias_act.py", line 16, in _get_plugin
return custom_ops.get_plugin(os.path.splitext(file)[0] + '.cu')
File "/home/fut/Downloads/stylegan2-NVlabs/dnnlib/tflib/custom_ops.py", line 156, in get_plugin
plugin = tf.load_op_library(bin_file)
File "/home/fut/miniconda3/envs/fut/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/fut/Downloads/stylegan2-NVlabs/dnnlib/tflib/_cudacache/fused_bias_act_09d1a1c1272fedad586c4a59ce52e74b.so: undefined symbol: _ZN10tensorflow12OpDefBuilder5InputESs

environment:
ubuntu18.04
tensorflow-gpu==1.14.0
I reinstall tensorflow-gpu,but the same error like before.How can solve this problem?

Hi and thanks for sharing you code ;)

Not really an issue really but I was wondering which model you've used to generate the fake portraits in the readme:

https://github.com/skyflynil/stylegan2/blob/master/fake_art_portrait.jpg

After successfully training StyleGAN 2 on a dataset of (1024, 768), I tried using the network for projecting real images into latent space. Unfortunately I'm getting an error as shown below. Traceback (most recent call last):
File "run_projector.py", line 146, in
main()
File "run_projector.py", line 141, in main
dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
File "/home/dayu/D/src2000/stylegan2_skyfly/dnnlib/submission/submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "/home/dayu/D/src2000/stylegan2_skyfly/dnnlib/submission/internal/local.py", line 22, in submit
return run_wrapper(submit_config)
File "/home/dayu/D/src2000/stylegan2_skyfly/dnnlib/submission/submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "/home/dayu/D/src2000/stylegan2_skyfly/run_projector.py", line 59, in project_real_images
proj.set_network(Gs)
File "/home/dayu/D/src2000/stylegan2_skyfly/projector.py", line 103, in set_network
proc_images_expr = tf.reduce_mean(tf.reshape(proc_images_expr, [-1, sh[1], sh[2] // factor, factor, sh[2] // factor, factor]), axis=[3,5])
File "/home/dayu/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 7715, in reshape
"Reshape", tensor=tensor, shape=shape, name=name)
File "/home/dayu/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/dayu/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/dayu/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/dayu/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2027, in init
control_input_ops)
File "/home/dayu/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1867, in _create_c_op
raise ValueError(str(e))
ValueError: Dimension size must be evenly divisible by 3145728 but is 2359296 for 'Reshape' (op: 'Reshape') with input shapes: [1,3,1024,768], [6] and with input tensors computed as partial shapes: input[1] = [?,3,256,4,256,4].

Trying to use run_training.py as so:
!python run_training.py --num-gpus=1 --data-dir=dataset --config=config-f --dataset=blows --mirror-augment=false --metric=none --total-kimg=20000 --result-dir="/content/drive/My Drive/stylegan2/results" gives me the following error:

Local submit - run_dir: /content/drive/My Drive/stylegan2/results/00026-stylegan2-blows-1gpu-config-f
dnnlib: Running training.training_loop.training_loop() on localhost...
Streaming data using training.dataset.TFRecordDataset...
Traceback (most recent call last):
  File "run_training.py", line 209, in <module>
    main()
  File "run_training.py", line 204, in main
    run(**vars(args))
  File "run_training.py", line 129, in run
    dnnlib.submit_run(**kwargs)
  File "/content/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run
    return farm.submit(submit_config, host_run_dir)
  File "/content/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit
    return run_wrapper(submit_config)
  File "/content/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper
    run_func_obj(**submit_config.run_func_kwargs)
  File "/content/stylegan2/training/training_loop.py", line 156, in training_loop
    training_set = dataset.load_dataset(data_dir=dnnlib.convert_path(data_dir), verbose=True, **dataset_args)
  File "/content/stylegan2/training/dataset.py", line 239, in load_dataset
    dataset = dnnlib.util.get_obj_by_name(class_name)(**adjusted_kwargs)
  File "/content/stylegan2/training/dataset.py", line 167, in __init__
    dset = dset.map(parse_tfrecord_tf_raw, num_parallel_calls=num_threads)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 1913, in map
    self, map_func, num_parallel_calls, preserve_cardinality=False))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3472, in __init__
    use_legacy_function=use_legacy_function)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2713, in __init__
    self._function = wrapper_fn._get_concrete_function_internal()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1853, in _get_concrete_function_internal
    *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1847, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2147, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2038, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2707, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2652, in _wrapper_helper
    ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in converted code:

    /content/stylegan2/training/dataset.py:27 parse_tfrecord_tf_raw  *
        features = tf.parse_single_example(
    /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py:1019 parse_single_example
        serialized, features, example_names, name
    /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py:1063 parse_single_example_v2_unoptimized
        return parse_single_example_v2(serialized, features, name)
    /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py:2093 parse_single_example_v2
        dense_defaults, dense_shapes, name)
    /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/parsing_ops.py:2210 _parse_single_example_v2_raw
        name=name)
    /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_parsing_ops.py:1201 parse_single_example
        dense_shapes=dense_shapes, name=name)
    /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py:551 _apply_op_helper
        (prefix, dtypes.as_dtype(input_arg.type).name))

    TypeError: Input 'serialized' of 'ParseSingleExample' Op has type uint8 that does not match expected type of string.

yet running the following returns no errors:

features = tf.parse_single_example(
        '/content/stylegan2/dataset/blows/blows-r07.tfrecords',
        features={
            "shape": tf.FixedLenFeature([3], tf.int64),
            "img": tf.FixedLenFeature([], tf.string),
        },
    )

Many thanks @skyflynil for sharing such an amazing work.
I would like to ask you if you have a working version of projector.py file for non-square images.

projector.py throwing exception for reshape operation at line: 122 if the model is trained for non-square images

When I have commented the lines: 122-123
#v = tf.reshape(v, [1, 1, sz//2, 2, sz//2, 2]) # Downscale
#v = tf.reduce_mean(v, axis=[3, 5])
it worked but gave wrong results.

Any suggestion for how to set a correct numbers in:
v = tf.reshape(v, [1, 1, sz//2, 2, sz//2, 2]) # Downscale
Thanks

first off, thanks for making this. I’m really excited to see if I can get hi-res non-squares working.

I have a set of 1280x768 images (10x6,n=7) but get the following error when trying to train.

(0) Invalid argument: Cannot batch tensors with different shapes in component 0. First element had shape [3,1280,685] and element 1 had shape [3,1280,768].
	 [[node GPU0/DataFetch/IteratorGetNext (defined at /root/stylegan2/training/dataset.py:193) ]]
	 [[GPU0/DataFetch/IteratorGetNext/_4117]]
  (1) Invalid argument: Cannot batch tensors with different shapes in component 0. First element had shape [3,1280,685] and element 1 had shape [3,1280,768].

I’ve double-checked and all images seem to be 1280x768. Any other thoughts on what could be going on here?

full log attached
log.txt

I'd like to resize my images before running dataset_tool.py. Should I do it within create_from_images_raw ? what would be the most optimal way to do it ? thanks !!

After successfully training StyleGAN 2 on a dataset of B&W images (1, 128, 128), I tried using the network for projecting real images into latent space. Unfortunately it seems like it was only coded with RGB in mind because I'm getting an error related to channels (1 vs. 3), as shown below. Unfortunately I can't find a place in the code where the 3 channels are defined. Is there any way of modifying the projector for B&W images?

Local submit - run_dir: results/00000-project-real-images
dnnlib: Running run_projector.project_real_images() on localhost...
Loading networks from "/content/stylegan2/network-snapshot-000215.pkl"...
Setting up TensorFlow plugin "fused_bias_act.cu": Preprocessing... Compiling... Loading... Done.
Setting up TensorFlow plugin "upfirdn_2d.cu": Preprocessing... Compiling... Loading... Done.
Downloading http://d36zk2xti64re0.cloudfront.net/stylegan1/networks/metrics/vgg16_zhang_perceptual.pkl ... done
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 642, in set_shape
    unknown_shape)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 1 in both shapes must be equal, but are 1 and 3. Shapes are [1,1,128,128] and [?,3,?,?].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_projector.py", line 146, in <module>
    main()
  File "run_projector.py", line 141, in main
    dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
  File "/content/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run
    return farm.submit(submit_config, host_run_dir)
  File "/content/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit
    return run_wrapper(submit_config)
  File "/content/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper
    run_func_obj(**submit_config.run_func_kwargs)
  File "/content/stylegan2/run_projector.py", line 59, in project_real_images
    proj.set_network(Gs)
  File "/content/stylegan2/projector.py", line 114, in set_network
    self._dist = self._lpips.get_output_for(proc_images_expr, self._target_images_var)
  File "/content/stylegan2/dnnlib/tflib/network.py", line 221, in get_output_for
    out_expr = self._build_func(*final_inputs, **build_kwargs)
  File "<string>", line 132, in lpips_network
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 645, in set_shape
    raise ValueError(str(e))
ValueError: Dimension 1 in both shapes must be equal, but are 1 and 3. Shapes are [1,1,128,128] and [?,3,?,?].

Thanks for some nice improvements!

In the readme there are some outputs for what seems to be wikiart portraits in 768x512. any chance of accessing that model for some transfer learning?
thanks!

hello there, i got your fork running on colab - semi fine.
like i said in the titled, the training stop after 4 ticks

tick 0 kimg 0.1 lod 0.00 minibatch 32 time 58s sec/tick 57.7 sec/kimg 450.76 maintenance 0.0 gpumem 5.1
tick 1 kimg 6.1 lod 0.00 minibatch 32 time 12m 16s sec/tick 648.1 sec/kimg 107.73 maintenance 30.5 gpumem 5.1
tick 2 kimg 12.2 lod 0.00 minibatch 32 time 23m 19s sec/tick 644.3 sec/kimg 107.10 maintenance 18.0 gpumem 5.1
tick 3 kimg 18.2 lod 0.00 minibatch 32 time 34m 16s sec/tick 652.4 sec/kimg 108.45 maintenance 5.1 gpumem 5.1
^C

^c like a keyboard interrupt.. but i didn't give such a command

Hello, I have been trying to train with my custom dataset (48k, 640x640) and the follwing error appears:

"Cannot batch tensors with different shapes in component 0. First element had shape [3,640,640] and element 1 had shape [1,640,640]"

I have used "--min-h=5 --min-w=5 --res-log2=7".

That first mismatching number is the batch size, isn't it? How could I solve this?

Thanks in advance!

Hi, congrats on this code! I'm training a model with 768x1280 dataset with this command:

nohup python run_training.py --num-gpus=1 --data-dir=./dataset --config=config-f --dataset=stainedglass1 --mirror-augment=true --metric=none --total-kimg=20000 --min-h=5 --min-w=3 --res-log2=8

I'm then getting this error after the second tick running this fork:

Traceback (most recent call last): File "run_training.py", line 218, in <module> main() File "run_training.py", line 213, in main run(**vars(args)) File "run_training.py", line 136, in run dnnlib.submit_run(**kwargs) File "/home/rmbwalsh/stylegan-skyflynil/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run return farm.submit(submit_config, host_run_dir) File "/home/rmbwalsh/stylegan-skyflynil/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit return run_wrapper(submit_config) File "/home/rmbwalsh/stylegan-skyflynil/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper run_func_obj(**submit_config.run_func_kwargs) File "/home/rmbwalsh/stylegan-skyflynil/stylegan2/training/training_loop.py", line 349, in training_loop grid_fakes = Gs.run(grid_latents, grid_labels, is_validation=True, minibatch_size=sched.minibatch_gpu) File "/home/rmbwalsh/stylegan-skyflynil/stylegan2/dnnlib/tflib/network.py", line 433, in run out_arrays = [np.empty([num_items] + expr.shape.as_list()[1:], expr.dtype.name) for expr in out_expr] File "/home/rmbwalsh/stylegan-skyflynil/stylegan2/dnnlib/tflib/network.py", line 433, in <listcomp> out_arrays = [np.empty([num_items] + expr.shape.as_list()[1:], expr.dtype.name) for expr in out_expr] MemoryError

I also got this error on another attempt:

MemoryError: Unable to allocate 450. MiB for an array with shape (40, 3, 1280, 768) and data type float32

Anyone had anything similar?

Hi together,
working under Windows10, Anocanda with tensofllow 1.14.0
I tried to run the generator-samples, but got errors opening the pkl.
python run_generator.py generate-images --network=gdrive:networks/stylegan2-ffhq-config-f.pkl --seeds=6600-6625 --truncation-psi=0.5
Error:
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 62: character maps to

Googled a lot, tried many encodings (latin1, utf8, some more...) , errors="ignore ", errors="replace". Or: open(path_or_url, 'rb') or 'r' ...Nothing worked for me.
@ windows-users: anyone got it ro run?

Any help would be appreciated :-)

Hey absolutely love the functionalities of this repo!! Thx so much!

However, I'm getting issues when running this on my local RTX 2080 Ti machine..
All the training ops run fine, except for the G_reg_op

The weird thing is that I just tested with the code from the official StyleGAN2 repo, and there everything seems to work perfectly (including G_reg_op)...

To summarize, the core of the error seems to originate in the custom cuda functions for upsampling:

• I have tested that nvcc test_nvcc.cu -o test_nvcc -run runs correctly
• Running Cuda 10.0, TF 1.14.0 and Python 3.6
• When I switch off lazy regularization, all runs fine again
• Running a config-f model on 1024x1024 res with (8, 8, res_log2 = 7) setting (is that the appropriate way of doing 1024x1024, or would you recommend (4,4, res_log2 = 8) ?

Here's the full traceback:

``dnnlib: Running training.training_loop.training_loop() on localhost...
Streaming data using training.dataset.TFRecordDataset...
####################################################
TFREC dir: datasets/fractal_tf_rec
['fractal_tf_rec-r07.tfrecords']
####################################################
Dataset shape = [3, 1024, 1024]
Dynamic range = [0, 255]
Label size = 0
Constructing networks...
Setting up TensorFlow plugin "fused_bias_act.cu": Preprocessing... Loading... Done.
Setting up TensorFlow plugin "upfirdn_2d.cu": Preprocessing... Loading... Done.

G                               Params    OutputShape          WeightShape     
---                             ---       ---                  ---             
latents_in                      -         (?, 512)             -               
labels_in                       -         (?, 0)               -               
lod                             -         ()                   -               
dlatent_avg                     -         (512,)               -               
G_mapping/latents_in            -         (?, 512)             -               
G_mapping/labels_in             -         (?, 0)               -               
G_mapping/Normalize             -         (?, 512)             -               
G_mapping/Dense0                262656    (?, 512)             (512, 512)      
G_mapping/Dense1                262656    (?, 512)             (512, 512)      
G_mapping/Dense2                262656    (?, 512)             (512, 512)      
G_mapping/Dense3                262656    (?, 512)             (512, 512)      
G_mapping/Dense4                262656    (?, 512)             (512, 512)      
G_mapping/Dense5                262656    (?, 512)             (512, 512)      
G_mapping/Dense6                262656    (?, 512)             (512, 512)      
G_mapping/Dense7                262656    (?, 512)             (512, 512)      
G_mapping/Broadcast             -         (?, 16, 512)         -               
G_mapping/dlatents_out          -         (?, 16, 512)         -               
Truncation/Lerp                 -         (?, 16, 512)         -               
G_synthesis/dlatents_in         -         (?, 16, 512)         -               
G_synthesis/8x8/Const           32768     (?, 512, 8, 8)       (1, 512, 8, 8)  
G_synthesis/8x8/Conv            2622465   (?, 512, 8, 8)       (3, 3, 512, 512)
G_synthesis/8x8/ToRGB           264195    (?, 3, 8, 8)         (1, 1, 512, 3)  
G_synthesis/16x16/Conv0_up      2622465   (?, 512, 16, 16)     (3, 3, 512, 512)
G_synthesis/16x16/Conv1         2622465   (?, 512, 16, 16)     (3, 3, 512, 512)
G_synthesis/16x16/Upsample      -         (?, 3, 16, 16)       -               
G_synthesis/16x16/ToRGB         264195    (?, 3, 16, 16)       (1, 1, 512, 3)  
G_synthesis/32x32/Conv0_up      2622465   (?, 512, 32, 32)     (3, 3, 512, 512)
G_synthesis/32x32/Conv1         2622465   (?, 512, 32, 32)     (3, 3, 512, 512)
G_synthesis/32x32/Upsample      -         (?, 3, 32, 32)       -               
G_synthesis/32x32/ToRGB         264195    (?, 3, 32, 32)       (1, 1, 512, 3)  
G_synthesis/64x64/Conv0_up      2622465   (?, 512, 64, 64)     (3, 3, 512, 512)
G_synthesis/64x64/Conv1         2622465   (?, 512, 64, 64)     (3, 3, 512, 512)
G_synthesis/64x64/Upsample      -         (?, 3, 64, 64)       -               
G_synthesis/64x64/ToRGB         264195    (?, 3, 64, 64)       (1, 1, 512, 3)  
G_synthesis/128x128/Conv0_up    1442561   (?, 256, 128, 128)   (3, 3, 512, 256)
G_synthesis/128x128/Conv1       721409    (?, 256, 128, 128)   (3, 3, 256, 256)
G_synthesis/128x128/Upsample    -         (?, 3, 128, 128)     -               
G_synthesis/128x128/ToRGB       132099    (?, 3, 128, 128)     (1, 1, 256, 3)  
G_synthesis/256x256/Conv0_up    426369    (?, 128, 256, 256)   (3, 3, 256, 128)
G_synthesis/256x256/Conv1       213249    (?, 128, 256, 256)   (3, 3, 128, 128)
G_synthesis/256x256/Upsample    -         (?, 3, 256, 256)     -               
G_synthesis/256x256/ToRGB       66051     (?, 3, 256, 256)     (1, 1, 128, 3)  
G_synthesis/512x512/Conv0_up    139457    (?, 64, 512, 512)    (3, 3, 128, 64) 
G_synthesis/512x512/Conv1       69761     (?, 64, 512, 512)    (3, 3, 64, 64)  
G_synthesis/512x512/Upsample    -         (?, 3, 512, 512)     -               
G_synthesis/512x512/ToRGB       33027     (?, 3, 512, 512)     (1, 1, 64, 3)   
G_synthesis/1024x1024/Conv0_up  51297     (?, 32, 1024, 1024)  (3, 3, 64, 32)  
G_synthesis/1024x1024/Conv1     25665     (?, 32, 1024, 1024)  (3, 3, 32, 32)  
G_synthesis/1024x1024/Upsample  -         (?, 3, 1024, 1024)   -               
G_synthesis/1024x1024/ToRGB     16515     (?, 3, 1024, 1024)   (1, 1, 32, 3)   
G_synthesis/images_out          -         (?, 3, 1024, 1024)   -               
G_synthesis/noise0              -         (1, 1, 8, 8)         -               
G_synthesis/noise1              -         (1, 1, 16, 16)       -               
G_synthesis/noise2              -         (1, 1, 16, 16)       -               
G_synthesis/noise3              -         (1, 1, 32, 32)       -               
G_synthesis/noise4              -         (1, 1, 32, 32)       -               
G_synthesis/noise5              -         (1, 1, 64, 64)       -               
G_synthesis/noise6              -         (1, 1, 64, 64)       -               
G_synthesis/noise7              -         (1, 1, 128, 128)     -               
G_synthesis/noise8              -         (1, 1, 128, 128)     -               
G_synthesis/noise9              -         (1, 1, 256, 256)     -               
G_synthesis/noise10             -         (1, 1, 256, 256)     -               
G_synthesis/noise11             -         (1, 1, 512, 512)     -               
G_synthesis/noise12             -         (1, 1, 512, 512)     -               
G_synthesis/noise13             -         (1, 1, 1024, 1024)   -               
G_synthesis/noise14             -         (1, 1, 1024, 1024)   -               
images_out                      -         (?, 3, 1024, 1024)   -               
---                             ---       ---                  ---             
Total                           24885511                                       


D                     Params    OutputShape          WeightShape     
---                   ---       ---                  ---             
images_in             -         (?, 3, 1024, 1024)   -               
labels_in             -         (?, 0)               -               
1024x1024/FromRGB     128       (?, 32, 1024, 1024)  (1, 1, 3, 32)   
1024x1024/Conv0       9248      (?, 32, 1024, 1024)  (3, 3, 32, 32)  
1024x1024/Conv1_down  18496     (?, 64, 512, 512)    (3, 3, 32, 64)  
1024x1024/Skip        2048      (?, 64, 512, 512)    (1, 1, 32, 64)  
512x512/Conv0         36928     (?, 64, 512, 512)    (3, 3, 64, 64)  
512x512/Conv1_down    73856     (?, 128, 256, 256)   (3, 3, 64, 128) 
512x512/Skip          8192      (?, 128, 256, 256)   (1, 1, 64, 128) 
256x256/Conv0         147584    (?, 128, 256, 256)   (3, 3, 128, 128)
256x256/Conv1_down    295168    (?, 256, 128, 128)   (3, 3, 128, 256)
256x256/Skip          32768     (?, 256, 128, 128)   (1, 1, 128, 256)
128x128/Conv0         590080    (?, 256, 128, 128)   (3, 3, 256, 256)
128x128/Conv1_down    1180160   (?, 512, 64, 64)     (3, 3, 256, 512)
128x128/Skip          131072    (?, 512, 64, 64)     (1, 1, 256, 512)
64x64/Conv0           2359808   (?, 512, 64, 64)     (3, 3, 512, 512)
64x64/Conv1_down      2359808   (?, 512, 32, 32)     (3, 3, 512, 512)
64x64/Skip            262144    (?, 512, 32, 32)     (1, 1, 512, 512)
32x32/Conv0           2359808   (?, 512, 32, 32)     (3, 3, 512, 512)
32x32/Conv1_down      2359808   (?, 512, 16, 16)     (3, 3, 512, 512)
32x32/Skip            262144    (?, 512, 16, 16)     (1, 1, 512, 512)
16x16/Conv0           2359808   (?, 512, 16, 16)     (3, 3, 512, 512)
16x16/Conv1_down      2359808   (?, 512, 8, 8)       (3, 3, 512, 512)
16x16/Skip            262144    (?, 512, 8, 8)       (1, 1, 512, 512)
8x8/MinibatchStddev   -         (?, 513, 8, 8)       -               
8x8/Conv              2364416   (?, 512, 8, 8)       (3, 3, 513, 512)
8x8/Dense0            16777728  (?, 512)             (32768, 512)    
Output                513       (?, 1)               (512, 1)        
scores_out            -         (?, 1)               -               
---                   ---       ---                  ---             
Total                 36613665                                       

Building TensorFlow graph...
Initializing logs...
Training for 20000 kimg...

Running with: minibatch_size_in: 64.00 -- minibatch_gpu_in: 1.00

Running with gradient accumulation
Trying G_reg_op...

Traceback (most recent call last):
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: cudaErrorInvalidConfiguration
	 [[{{node GPU0/G_loss/PathReg/G/G_synthesis/16x16/Upsample/UpFirDn2D}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_training.py", line 216, in <module>
    main()
  File "run_training.py", line 211, in main
    run(**vars(args))
  File "run_training.py", line 136, in run
    dnnlib.submit_run(**kwargs)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run
    return farm.submit(submit_config, host_run_dir)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit
    return run_wrapper(submit_config)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper
    run_func_obj(**submit_config.run_func_kwargs)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/training/training_loop.py", line 347, in training_loop
    tflib.run(G_reg_op, feed_dict)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/tfutil.py", line 31, in run
    return tf.get_default_session().run(*args, **kwargs)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: cudaErrorInvalidConfiguration
	 [[node GPU0/G_loss/PathReg/G/G_synthesis/16x16/Upsample/UpFirDn2D (defined at <string>:110) ]]

Errors may have originated from an input operation.
Input Source operations connected to node GPU0/G_loss/PathReg/G/G_synthesis/16x16/Upsample/UpFirDn2D:
 GPU0/G_loss/PathReg/G/G_synthesis/16x16/Upsample/Reshape (defined at /home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/ops/upfirdn_2d.py:358)	
 GPU0/G_loss/PathReg/G/G_synthesis/16x16/Upsample/Const (defined at /home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/ops/upfirdn_2d.py:123)

Original stack trace for 'GPU0/G_loss/PathReg/G/G_synthesis/16x16/Upsample/UpFirDn2D':
  File "run_training.py", line 216, in <module>
    main()
  File "run_training.py", line 211, in main
    run(**vars(args))
  File "run_training.py", line 136, in run
    dnnlib.submit_run(**kwargs)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run
    return farm.submit(submit_config, host_run_dir)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit
    return run_wrapper(submit_config)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper
    run_func_obj(**submit_config.run_func_kwargs)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/training/training_loop.py", line 256, in training_loop
    G_loss, G_reg = dnnlib.util.call_func_by_name(G=G_gpu, D=D_gpu, opt=G_opt, training_set=training_set, minibatch_size=minibatch_gpu_in, **G_loss_args)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/util.py", line 256, in call_func_by_name
    return func_obj(*args, **kwargs)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/training/loss.py", line 164, in G_logistic_ns_pathreg
    fake_images_out, fake_dlatents_out = G.get_output_for(pl_latents, pl_labels, is_training=True, return_dlatents=True)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/network.py", line 221, in get_output_for
    out_expr = self._build_func(*final_inputs, **build_kwargs)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/training/networks_stylegan2.py", line 238, in G_main
    images_out = components.synthesis.get_output_for(dlatents, is_training=is_training, force_clean_graph=is_template_graph, **kwargs)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/network.py", line 221, in get_output_for
    out_expr = self._build_func(*final_inputs, **build_kwargs)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/training/networks_stylegan2.py", line 544, in G_synthesis_stylegan2
    y = upsample(y)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/training/networks_stylegan2.py", line 499, in upsample
    return upsample_2d(y, k=resample_kernel)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/ops/upfirdn_2d.py", line 198, in upsample_2d
    return _simple_upfirdn_2d(x, k, up=factor, pad0=(p+1)//2+factor-1, pad1=p//2, data_format=data_format, impl=impl)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/ops/upfirdn_2d.py", line 359, in _simple_upfirdn_2d
    y = upfirdn_2d(y, k, upx=up, upy=up, downx=down, downy=down, padx0=pad0, padx1=pad1, pady0=pad0, pady1=pad1, impl=impl)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/ops/upfirdn_2d.py", line 62, in upfirdn_2d
    return impl_dict[impl](x=x, k=k, upx=upx, upy=upy, downx=downx, downy=downy, padx0=padx0, padx1=padx1, pady0=pady0, pady1=pady1)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/ops/upfirdn_2d.py", line 140, in _upfirdn_2d_cuda
    return func(x)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/ops/custom_gradient.py", line 162, in decorated
    return _graph_mode_decorator(f, *args, **kwargs)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/ops/custom_gradient.py", line 183, in _graph_mode_decorator
    result, grad_fn = f(*args)
  File "/home/rednax/Desktop/music_vr/StyleGAN_training/stylegan2/dnnlib/tflib/ops/upfirdn_2d.py", line 132, in func
    y = _get_plugin().up_fir_dn2d(x=x, k=kc, upx=upx, upy=upy, downx=downx, downy=downy, padx0=padx0, padx1=padx1, pady0=pady0, pady1=pady1)
  File "<string>", line 110, in up_fir_dn2d
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/home/rednax/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()
``

I converted a dataset of 912 JPEGs, measuring 1408x1152 (hxw) using this command:
python dataset_tool.py create_from_images --res_log2=7 ./datasets/shots /content/stylegan2-colab/stylegan2/raw_images/shots/

Then, I attempted to begin training with this command:
python run_training.py --num-gpus=1 --data-dir=./datasets --config=config-f --dataset=shots --mirror-augment=true --metric=none --total-kimg=1000 --min-h=11 --min-w=9 --res-log2=7 --result-dir="/content/stylegan2-colab/stylegan2/results"

But I received the following error almost immediately:

File "/content/stylegan2-colab/stylegan2/training/dataset.py", line 56, in parse_tfrecord_np_raw
    0
IndexError: list index (0) out of range

Did I do something wrong?

My image is 1024 * 1024, and my parameters are
!python run_training.py --num-gpus=1 --data-dir=../stylegan2/dataset --config=config-f --dataset=rice0 --mirror-augment=true --metric=none --total-kimg=40000 --min-h=8 --min-w=8 --res-log2=7

error:
Traceback (most recent call last):
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Cannot batch tensors with different shapes in component 0. First element had shape [3,1024,1024] and element 1 had shape [3,3840,2699].
[[{{node GPU0/DataFetch/IteratorGetNext}}]]
(1) Invalid argument: Cannot batch tensors with different shapes in component 0. First element had shape [3,1024,1024] and element 1 had shape [3,3840,2699].
[[{{node GPU0/DataFetch/IteratorGetNext}}]]
[[GPU0/DataFetch/IteratorGetNext/_4567]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run_training.py", line 218, in
main()
File "run_training.py", line 213, in main
run(**vars(args))
File "run_training.py", line 136, in run
dnnlib.submit_run(**kwargs)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit
return run_wrapper(submit_config)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/training/training_loop.py", line 318, in training_loop
tflib.run(data_fetch_op, feed_dict_d)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/dnnlib/tflib/tfutil.py", line 31, in run
return tf.get_default_session().run(*args, **kwargs)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Cannot batch tensors with different shapes in component 0. First element had shape [3,1024,1024] and element 1 had shape [3,3840,2699].
[[node GPU0/DataFetch/IteratorGetNext (defined at /tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Invalid argument: Cannot batch tensors with different shapes in component 0. First element had shape [3,1024,1024] and element 1 had shape [3,3840,2699].
[[node GPU0/DataFetch/IteratorGetNext (defined at /tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py:1748) ]]
[[GPU0/DataFetch/IteratorGetNext/_4567]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'GPU0/DataFetch/IteratorGetNext':
File "run_training.py", line 218, in
main()
File "run_training.py", line 213, in main
run(**vars(args))
File "run_training.py", line 136, in run
dnnlib.submit_run(**kwargs)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit
return run_wrapper(submit_config)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "/content/drive/My Drive/Colab Notebooks/stylegan2/training/training_loop.py", line 216, in training_loop
reals_write, labels_write = training_set.get_minibatch_tf()
File "/content/drive/My Drive/Colab Notebooks/stylegan2/training/dataset.py", line 198, in get_minibatch_tf
return self._tf_iterator.get_next()
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
name=name)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next
output_shapes=output_shapes, name=name)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/tensorflow-1.15.2/python3.6/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

Hi,

I get this message when I try to run training on my custom dataset.

ValueError: Dimension 2 in both shapes must be equal, but are 1024 and 1280.

I know all my images are 1024x1024 but the error message seems to suggest otherwise. And the same images work fine on the original stylegan2, so not sure where this is going wrong.

Thanks

When I run $ python run_training.py --num-gpus=1 --data-dir=/data/ --config=config-f --dataset=dataset --mirror-augment=true --metric=none --total-kimg=20000 --result-dir="~/chenyulan/data/results", I get an error as follow:
Setting up TensorFlow plugin "fused_bias_act.cu": Preprocessing... Compiling... Loading... Failed!
Traceback (most recent call last):
File "run_training.py", line 230, in
main()
File "run_training.py", line 225, in main
run(**vars(args))
File "run_training.py", line 144, in run
dnnlib.submit_run(**kwargs)
File "/root/stylegan2_train/dnnlib/submission/submit.py", line 343, in submit_run
return farm.submit(submit_config, host_run_dir)
File "/root/stylegan2_train/dnnlib/submission/internal/local.py", line 22, in submit
return run_wrapper(submit_config)
File "/root/stylegan2_train/dnnlib/submission/submit.py", line 280, in run_wrapper
run_func_obj(**submit_config.run_func_kwargs)
File "/root/stylegan2_train/training/training_loop.py", line 179, in training_loop
G = tflib.Network('G', num_channels=training_set.shape[0], resolution=training_set.shape[1], label_size=training_set.label_size, **G_args)
File "/root/stylegan2_train/dnnlib/tflib/network.py", line 97, in init
self._init_graph()
File "/root/stylegan2_train/dnnlib/tflib/network.py", line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File "/root/stylegan2_train/training/networks_stylegan2.py", line 288, in G_main
components.synthesis = tflib.Network('G_synthesis', func_name=globals()[synthesis_func], **kwargs)
File "/root/stylegan2_train/dnnlib/tflib/network.py", line 97, in init
self._init_graph()
File "/root/stylegan2_train/dnnlib/tflib/network.py", line 154, in _init_graph
out_expr = self._build_func(*self.input_templates, **build_kwargs)
File "/root/stylegan2_train/training/networks_stylegan2.py", line 641, in G_synthesis_stylegan2
x = layer(x, layer_idx=0, fmaps=nf(1), kernel=3)
File "/root/stylegan2_train/training/networks_stylegan2.py", line 565, in layer
x = modulated_conv2d_layer(x, dlatents_in[:, layer_idx], fmaps=fmaps, kernel=kernel, up=up, resample_kernel=resample_kernel, fused_modconv=fused_modconv)
File "/root/stylegan2_train/training/networks_stylegan2.py", line 100, in modulated_conv2d_layer
s = apply_bias_act(s, bias_var=mod_bias_var) + 1 # [BI] Add bias (initially 1).
File "/root/stylegan2_train/training/networks_stylegan2.py", line 69, in apply_bias_act
return fused_bias_act(x, b=tf.cast(b, x.dtype), act=act, alpha=alpha, gain=gain)
File "/root/stylegan2_train/dnnlib/tflib/ops/fused_bias_act.py", line 68, in fused_bias_act
return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
File "/root/stylegan2_train/dnnlib/tflib/ops/fused_bias_act.py", line 122, in _fused_bias_act_cuda
cuda_kernel = _get_plugin().fused_bias_act
File "/root/stylegan2_train/dnnlib/tflib/ops/fused_bias_act.py", line 16, in _get_plugin
return custom_ops.get_plugin(os.path.splitext(file)[0] + '.cu')
File "/root/stylegan2_train/dnnlib/tflib/custom_ops.py", line 160, in get_plugin
plugin = tf.load_op_library(bin_file)
File "/root/anaconda3/envs/stylegan2/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /root/stylegan2_train/dnnlib/tflib/_cudacache/fused_bias_act_ec21d79f0dc288505704f796449a968e.so: undefined symbol: _ZN10tensorflow12OpDefBuilder6OutputESs

I try this way as fllow:
In file stylegan2/dnnlib/tflib/custom_ops.py, line 127:
change from
compile_opts += ’ --compiler-options \’-fPIC -D_GLIBCXX_USE_CXX11_ABI=0\’’
to
compile_opts += ’ --compiler-options \’-fPIC -D_GLIBCXX_USE_CXX11_ABI=1\’’
But ,It don't work

run : $ ldd root/stylegan2_train/dnnlib/tflib/_cudacache/fused_bias_act_ec21d79f0dc288505704f796449a968e.so':
linux-vdso.so.1 => (0x00007ffd1adb2000)
_pywrap_tensorflow_internal.so => not found
librt.so.1 => /usr/lib64/librt.so.1 (0x00007eff848ff000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007eff846e3000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007eff844df000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007eff84e2d000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00007eff841dd000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007eff83fc7000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007eff83c05000)
/lib64/ld-linux-x86-64.so.2 (0x00007eff84da4000)
environment:
python=3.6.12
tensorflow-gpu=1.14
cuda:10.0
cuDnn:7.6.5

Hi there,

Does anyone know why FID50k metric errors out?
When I set my --metrics=None, everything works fine(the network trains and generates pkls and images), but when I default to FID50k, I get this:

Traceback (most recent call last): File "run_training.py", line 223, in <module> main() File "run_training.py", line 218, in main run(**vars(args)) File "run_training.py", line 140, in run dnnlib.submit_run(**kwargs) File "./dnnlib/submission/submit.py", line 343, in submit_run return farm.submit(submit_config, host_run_dir) File "./dnnlib/submission/internal/local.py", line 22, in submit return run_wrapper(submit_config) File "./dnnlib/submission/submit.py", line 280, in run_wrapper run_func_obj(**submit_config.run_func_kwargs) File "./training/training_loop.py", line 377, in training_loop metrics.run(pkl, run_dir=dnnlib.make_run_dir_path(), data_dir=dnnlib.convert_path(data_dir), num_gpus=num_gpus, tf_config=tf_config) File "./metrics/metric_base.py", line 151, in run metric.run(*args, **kwargs) File "./metrics/metric_base.py", line 62, in run self._reset(network_pkl=network_pkl, run_dir=run_dir, data_dir=data_dir, dataset_args=dataset_args, mirror_augment=mirror_augment) File "./metrics/metric_base.py", line 39, in _reset self._dataset_obj.close() AttributeError: 'TFRecordDataset' object has no attribute 'close'

Any pointers?

Thanks in advance.

i'm attempting to create a dataset from images that are 320x448 (5x7 res-log2 = 6) but when i attempt to run the dataset tool with !python /content/stylegan2/dataset_tool.py create_from_images_raw --res_log2=6 ./dataset/CADGAN_320_448 '/content/CADGAN_320_448' i get this error

Loading images from "/content/CADGAN_320_448" detected 7114 images ... Shuffle the images... Creating dataset "./dataset/CADGAN_320_448" Added 0 images. Traceback (most recent call last): File "/content/stylegan2/dataset_tool.py", line 950, in <module> execute_cmdline(sys.argv) File "/content/stylegan2/dataset_tool.py", line 944, in execute_cmdline func(**vars(args)) File "/content/stylegan2/dataset_tool.py", line 710, in create_from_images_raw tfr.create_tfr_writer(img.shape) File "/content/stylegan2/dataset_tool.py", line 96, in create_tfr_writer assert self.shape[1] % (2 ** self.res_log2) == 0 AssertionError

i assume this has something to do with the size of the images, but i'm kind of at a loss otherwise.

The colab part is out of date! as tensorflow 1 is not supported anymore in google colab!

Hi,
does anyone know how to deal with that ?

raise RuntimeError("Invalid task name.  Probable reason: unacceptable characters in your submit_config.run_desc.  Task name must be accepted by the following regex: " + docker_valid_name_regex + ", got " + submit_config.task_name)
RuntimeError: Invalid task name.  Probable reason: unacceptable characters in your submit_config.run_desc.  Task name must be accepted by the following regex: ^[a-zA-Z0-9][a-zA-Z0-9_.-]+$, got jupyter-00003-stylegan2-raw_dataset/images-1gpu-config-f

Hello, may I ask how to keep the output image consistent every time?