Coder Social home page Coder Social logo

clip-mesh's Introduction

CLIP-Mesh

Official implementation of CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, Tiberiu Popa
SIGGRAPH ASIA 2022 | arXiv | Project page

CLIP-Mesh header image

Quickstart

Notebooks are unavilable working on restoring them:

Text to Mesh Multi Mesh Generation
Stylizing a Mesh Apply CLIP-Mesh to Human Models

Setup

Clone this repository recursively to get all submodules - use submodule update to get downstream submodules

git clone --recurse-submodules https://github.com/NasirKhalid24/CLIP-Mesh.git
cd CLIP-Mesh
git submodule update --init --recursive

Setup Conda environment and install packages

conda create -n clip-mesh python=3.7
conda activate clip-mesh
conda install pytorch==1.11.0 torchvision==0.12.0 cudatoolkit=10.2 -c pytorch

pip install -r requirements.txt

Install loop subdivison code and DALLE-2 (not that DALLE-2 is from an earlier commit so existing install may not work)

<!-- Install Loop Subdivision -->
cd loop_limitation
pip install .
cd ..

<!-- Install DALLE2 - Diffusion Prior -->
cd DALLE2-pytorch
pip install .
cd ..

<!-- Get DALLE2 Prior Weights -->
mkdir weights
wget https://huggingface.co/spaces/NasirKhalid24/Dalle2-Diffusion-Prior/resolve/main/larger-model.pth -O ./weights/model.pth

Usage

This repo comes with some configs that are passed to main.py using the --config flag

Any of the config paramaters can be overriden by passing them to as arguments to the main.py file so you can have a base .yml file with all your parameters and just update the text prompt to generate something new

An example would be using the given config file for single mesh generation single.yml

# Use all parameters in file
python main.py --config configs/single.yml      

# Use all parameters in file but change text prompt
python main.py --config configs/single.yml  --text_prompt "a hamburger"    

# Use all parameters in file but change text prompt, batch, texture resolution
python main.py \
--config configs/single.yml \
--text_prompt "a hamburger" \
--batch_size 5 \
--texture_resolution 1024

Tips, Tricks, FAQs etc

I recommend checking out the following document as it could answer any doubts that come up (will be updated regularly) - if you still have questions reach out @Nymarius_ or open an issue

clip-mesh's People

Contributors

nasirkhalid24 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

clip-mesh's Issues

Typo in the bibtex of the project page

Hi, I notice that there is a typo in the bibtex of the project page: journal is misspelled as joural. This will cause a citation problem when others want to cite CLIP-Mesh in their papers and copy the bibtex from the project page. I encountered this problem in a previous paper submission :(. I hope you can fix it soon.

Different mesh error

Getting errors when introduncing a new mesh for texture generation. The in-house meshes like spot.obj etc are working fine, but meshes generated from OpenAI's Point-E are not working well.

Missing LICENSE

I see you have no LICENSE file for this project. The default is copyright.

I would suggest releasing the code under the GPL-3.0-or-later or Apache-2.0 license so that others are encouraged to contribute changes back to your project.

Colab links not working

Hello the colab links are dead, I get the message: "Sorry, the file you have requested does not exist." could you please provide new ones, thank you in advance.

Training the diffusion prior

Hi, congratulations on your fantastic work! Could you please share the code for training the diffusion prior? I want to be more clear about some training details. Besides, I'm curious if the CLIP text embeddings and image embeddings are normalized when training the diffusion prior network. Thanks!

Diffusion Prior?

Does the diffusion prior need to come from the pretrained model in the README, or can we swap in any prior?

That is, assuming the prior works with whatever code is loading it. So for instance if I wanted to swap in another Latent Diffusion model than DALLE-2-pytorch.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.