Planet Diffusion

Fine-tuning stable diffusion to generate planet/moon textures.

Using Stable Diffusion XL + LoRA [v2]

Demos

Cherry-picked best of several generations with varying checkpoints, guidance scales, and seeds.

Prompt	Texture	Render
A planet with a surface of turquoise and gold, marked by large, glittering rivers of molten metal and bright, shining seas.
A small, rocky planet with a sandy, yellow surface, characterized by its large, winding canyons and massive, towering volcanoes
A planet with a surface covered in lush, green vegetation and large bodies of water, with a strikingly colorful atmosphere.
A frozen moon with a pristine white surface adorned with deep blue ice crevasses and towering ice peaks.
A dense, greenish gas giant, surrounded by a hazy, nebulous atmosphere, with faint, swirling bands visible across its surface.
A gas giant with a turbulent, stormy surface, displaying a mesmerizing swirl of teal, navy, and violet hues.

Training

Follow the instructions on sdxl-lora-planet-textures. You can also find several pre-trained models here.

LoRA enabled training on an NVIDIA 3090 Ti.

Inference

import torch
from diffusers import DiffusionPipeline, AutoencoderKL

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", vae=vae, torch_dtype=torch.float16, variant="fp16", use_safetensors=True)
pipe.load_lora_weights("sshh12/sdxl-lora-planet-textures")
pipe.to("cuda")

prompt = "A dwarf planet exhibiting a striking purple color, with a surface peppered with craters and towering ice formations"
negative_prompt = 'blurry, fuzzy, low resolution, cartoon, painting'

image = pipe(prompt=prompt, negative_prompt=negative_prompt, width=1024, height=512).images[0]
image

See scripts/generate_images.py for an example of advanced usage (including using an upscaler).

Using Stable Diffusion [v1]

Demos

Cherry-picked best-of-4. It tends to struggle with prompts involving oceans or continents as that's pretty overfit to Earth. Generally, this model is fairly overfit to existing objects in our solar system.

Prompt	Texture	Render
a red-ish moon full of large volcanos and craters. fictitious
a large gas giant with multi-color rainbow bands. fictitious
a dark grey cratered planet with large white icy poles. fictitious

Training

Generate a dataset with the scripts in the repo or use sshh12/planet-textures
Clone https://github.com/justinpinkney/stable-diffusion @ f1293f9795fda211d7fffdb84cd308424c2a184b and apply v1/stable-diffusion.patch
Train the model. I used a NVIDIA RTX A6000 on LambdaLabs. If you do everything correctly the first time, the expected cost is $12.

from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(repo_id="CompVis/stable-diffusion-v-1-4-original", filename="sd-v1-4-full-ema.ckpt")

!(python main.py \
    -t \
    --base ../v1/planet-diffusion.yaml \
    --gpus "1" \
    --scale_lr False \
    --num_nodes 1 \
    --check_val_every_n_epoch 10 \
    --finetune_from "$ckpt_path" \
    data.params.batch_size=1 \
    lightning.trainer.accumulate_grad_batches=8 \
    data.params.validation.params.n_gpus=1 \
)

Feel free to contact me (using GitHub issues) for the original weights or you run into issues setting this up.

Inference

!(python scripts/txt2img.py \
    --prompt 'your prompt here' \
    --outdir '../outputs/' \
    --H 512 --W 1024 \
    --n_samples 2 \
    --config '../v1/planet-diffusion.yaml' \
    --ckpt 'logs/2023-06-29T00-13-09_planet-diffusion/checkpoints/epoch=000249.ckpt')

sshh12 / planet-diffusion Goto Github PK