sentdex / gantheftauto Goto Github PK

Despite the names; these architectures couldn't really be more different. As such you would of course need to train a new model if you wanted this to work with GTA V, unfortunately.

Having said that however; the VQGAN presented by CompVis is highly effiicient and represents images with much better fidelity.

https://github.com/CompVis/taming-transformers

You can see their pretrained 1024 token model in action at DALLE-pytorch for just one example. It helped us significantly improve the image quality compared to using a traditional VAE.

I'm also still learning how this repo specifically works. With the VQGAN; you generally are given a single token which represents 16x16 pixel square samples.

One way that you may be able to get this to work would be to train a sort of multimodal transformer where you encode user input as one modality and visuals from the game as another. With the transformer approach you might also need to consider the time axis because you will be taking previous frames as input, is my understanding.

Unfortunately I'm still learning about a lot of this stuff; but I think this project is awesome for teaching purposes so I'd love to see it continue to grow and improve if possible!

There's a lot of new research in this area including the newly released Alias-Free-GAN paper - which could also significantly improve results while still maintaining a similar architecture.

Are there any plans to take advantage of these recent improvements? I assume the core team is still busy fixing things up and may not have even had time to consider any of this stuff - so "no" is certainly an acceptable answer :)

Errors may have originated from an input operation.
Input Source operations connected to node model/spectral_normalization_1/conv2d_1/Conv2D:
model/up_sampling2d/resize/ResizeNearestNeighbor (defined at \GANTheftAuto\upsample\upsample.py:17)

Not sure what the issue is, my GPU is a GTX 1070 & I have installed the latest drivers.

Because I'm stuck with Could not find module 'caffe2_nvrtc.dll' (or one of its dependencies).
And that sounds to me, after some googlin, that I'd need a CUDA capable GPU.

Hi,

What is the license for the GTA dataset by itself, as that was not collected by Nvidia?

Thanks!

When running /scripts/gtav_multi.sh I get RuntimeError:

[...]
optG_graphic, Include: graphics_renderer.output_layer.0.1.bias
setting up dataset
Start epoch 0...
Traceback (most recent call last):
  File "main_parallel.py", line 284, in <module>
    train_gamegan(opts.gpu, opts)
  File "main_parallel.py", line 190, in train_gamegan
    trainer.generator_trainstep(states, actions, warm_up=warm_up, epoch=epoch)
  File "/home/jupyter/GANTheftAuto/trainer.py", line 129, in generator_trainstep
    gout = self.netG(self.zdist, states, gen_actions, warm_up, train=train, epoch=epoch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/jupyter/GANTheftAuto/simulator_model/dynamics_engine.py", line 256, in forward
    hiddens, init_maps= self.run_warmup(zdist, states, actions, warm_up, train=train)
  File "/home/jupyter/GANTheftAuto/simulator_model/dynamics_engine.py", line 235, in run_warmup
    batch_size, prev_read_v, prev_alpha, M, zdist, step=i, force_sharp=force_sharp)
  File "/home/jupyter/GANTheftAuto/simulator_model/dynamics_engine.py", line 175, in run_step
    s = self.simple_enc(state)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/jupyter/GANTheftAuto/simulator_model/model_utils.py", line 30, in forward
    return tensor.view(self.size)
RuntimeError: shape '[-1, 3136]' is invalid for input of size 184320

This is after:

1. Removing logdir which was not recognized as an argument
2. Installing and validating pytorch 1.8.0, which seems to be the choice of this repo, so I doubt it is that.
3. Trying to adjust the resolution to match the input size (184320 = e.g. 640x288)
4. Searching thru the codebase for either number

What is strange is not that there is nothing, but rather a shape mismatch despite using the default parameters The strange thing is that the number that we are casting to (3136), does not clearly seem to relate to anything, but it is roughly on the same order as the resolution (80x48=3840).

This is made somewhat worse by the fact that

Hi, i got Gan Theft Auto to launch just fine, but it runs at 1 Frame a second. I figured it was because it wasn't properly detecting my GPU, and I found this:

2021-06-19 20:31:56.833152: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-06-19 20:31:56.833248: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1766] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

I tried everything and I'm not sure what could still be causing this? I have CUDA toolkit installed.

Not sure why but when running the demo bat file it seems to get stuck.

(base) C:\Users\lanzc\Documents\GitHub\GANTheftAuto>python inference.py --saved_model ./trained_models/gan_5_1_17.pt --data gtav:./data/gtav/gtagan_2_sample --inference_image_path ./data/gtav/2.png --show_base_images True --upsample_model ./trained_models/upsample---[_20]---[______3171]---[_____63420].h5
Adding attention layer in G at resolution 32
Adding attention layer in G at resolution 64
Param count for Gs initialized parameters: 29664579
Adding attention layer in D at resolution 64
Adding attention layer in D at resolution 32
Param count for Ds initialized parameters: 1081733

It stays here forever.

Test

Followed installation instructions step by step, these instructions to be exact

No errors during installation, running gtav_interface_demo.bat gives file not found error.

C:\Users\Martin\Desktop\ganTheftAuto\GANTheftAuto>scripts\gtav_inference_demo.bat

C:\Users\Martin\Desktop\ganTheftAuto\GANTheftAuto>python inference.py  --saved_model ./trained_models/gan_5_1_17.pt  --data gtav:./data/gtav/gtagan_2_sample  --inference_image_path ./data/gtav/2.png  --show_base_images True  --upsample_model ./trained_models/upsample---[_20]---[______3171]---[_____63420].h5
C:\Users\Martin\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\__init__.py
Traceback (most recent call last):
  File "C:\Users\Martin\Desktop\ganTheftAuto\GANTheftAuto\inference.py", line 25, in <module>
    ctypes.cdll.LoadLibrary('caffe2_nvrtc.dll')
  File "C:\Users\Martin\AppData\Local\Programs\Python\Python39\lib\ctypes\__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "C:\Users\Martin\AppData\Local\Programs\Python\Python39\lib\ctypes\__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'caffe2_nvrtc.dll' (or one of its dependencies). Try using the full path with constructor syntax.

Seems that this workaround for windows in interface.py does not work anymore because of newer version of pyTorch, thats my guess.

I have windows 10, dedicated gpu with drivers installed and up to date. nvcuda.dll present in C:/Windows/system32
If I am being stupid please let me know and feel free to close this issue.

max@pop-os:/media/max/Data1/GANTheftAuto$ ./scripts/gtav_inference_demo.sh
Traceback (most recent call last):
File "/media/max/Data1/GANTheftAuto/inference.py", line 273, in
inference(opts.gpu, opts)
File "/media/max/Data1/GANTheftAuto/inference.py", line 91, in inference
torch.cuda.set_device(gpu)
File "/home/max/.local/lib/python3.9/site-packages/torch/cuda/init.py", line 264, in set_device
torch._C._cuda_setDevice(device)
File "/home/max/.local/lib/python3.9/site-packages/torch/cuda/init.py", line 172, in _lazy_init
torch._C._cuda_init()
RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

Not mentioned tensorflow and tensorflow_addons for running gtav_inference_demo.sh as dependencies in requirements.txt