shahbuland / destinygan

Automatically generating Destiny 2 loot using StyleGAN2 ADA and SWAGAN

destinygan

Automatically generating Destiny 2 loot using StyleGAN2 with differntiable augmentation and GPT-3

Post-mortem

This is a project I haven't worked on in well over a year and could not complete. I've added to this repo everything I did accomplish for anyone that is interested in looking at the results I did end up getting or wants to make something else using the data scraper/model setup.

Things I did finish:

• A scraper that gets all gun flavor text and names
• A scraper that gets all gun images

Things that were WIP:

• Scraper for perk icons
• A model made specifically to generate small icons
• A contrastive model that made small icons FOR guns
• Final generated guns were meant to have their corner icon and a perk pool

What went wrong?

I suspect the dataset was not diverse enough for styleGAN to be trained properly. I proceeded to get trash results when using other peoples' implementations as well. The main issue was unavoidable mode collapse. This still led to me being able to generate some cool guns because different models gave different generations, but not having the style vector effect generated images kind of defeats the purpose of stylegan. I ended up just randomly sampling from different checkpoints to get generated sample diversity.

What went right?

The samples that did come out of the model still looked cool! Here are some samples: imgur album. Looking at it train and go from learning basic shapes to the actual weapons was quite cool as well: timelapse here. GPT-3 also did an amazing job of generating new names and flavor texts. The model also seemed to work quite well for the small icons: imgur album (this is throughout training so they'll get better as you scroll down). This probably further implies the issue WAS with the dataset and not my code.

Out of repo requirements:

• For text generation you need a GPT-3 key
• For scraping destiny data you need bungie API key
• Several of the model scripts require custom cuda kernels from the official stylegan repo from nvidia. Credit to them for that, I've putted it in a zip file for convenience: link. Just extract into model stylegan folder.
• The SWAGAN model probably isn't that well known but they reported much faster training so I implemented it here. Paper here if you want more info.

Training:

• I trained the model around the start of 2021. This is when the dataset was scraped as well, so it wouldn't have seen any new items that came out after that time frame
• It was trained many, many times (as I said, mode collapse, so different looking images are from different model checkpoints) on an RTX 3090.

Select Samples:

Fun fact: the names it uses that were already names of guns from the game were not in the prompt! This means GPT-3 has already read articles on Destiny and has memory of names of certain weapons from it. Certainly, it never saw No Land Beyond in my dataset, as it consists entirely of D2 guns.

The flavor text is pretty funny sometimes.
Apparently Cayde is just a guy.