Comments (9)
The matrices calculation in gradio_new.py is only for angle visualization purposes and is independent of the zero123 inference code, so I wouldn't use one to understand another.
Assuming you are asking zero123 inference conditioning, here's the code converting a pair of camera matices (input view and target view) into the 4-dimensional conditioning vector described in appendix section A:
zero123/zero123/ldm/data/simple.py
Lines 257 to 272 in f70ea8c
Hope it makes sense.
P.S. camera matrices over parameterized the transformation so there'll be more than one way to represent an RT operation. I think a better way to compare RT is to visualize the camera or to convert it to Euler or axis angle first.
from zero123.
The matrices calculation in gradio_new.py is only for angle visualization purposes and is independent of the zero123 inference code, so I wouldn't use one to understand another.
Assuming you are asking zero123 inference conditioning, here's the code converting a pair of camera matices (input view and target view) into the 4-dimensional conditioning vector described in appendix section A.
Hope it makes sense.
P.S. camera matrices over parameterized the transformation so there'll be more than one way to represent an RT operation. I think a better way to compare RT is to visualize the camera or to convert it to Euler or axis angle first.
if I use camera_R to calculate the rotation matric in gradio_new.py, is the camera_R the target image's rotation matric or not?
from zero123.
Sorry but I don't understand. What do you need camera_R, which is used only for visualization, for?
from zero123.
Sorry but I don't understand. What do you need camera_R, which is used only for visualization, for?
camera_R is not used only for visualization. I want to input a single RGB image and then synthesize an image from a specified camera viewpoint. if I give many specified camera viewpoints, I can get many target views. Then, I utilize the target image and corresponding extrinsic matric(eg: camera_R) as a new dataset. So I want to know if the camera_R in gradio_new.py is the target image rotation matric. if yes, I can calculate the translation matric T in gradio_new.py.
from zero123.
If you want to generate a novel view based on a sampled target viewpoint described by an RT matrix, assuming your input viewpoint is cond_RT, you can sample a target viewpoint target_RT, and use the get_T function to get d_T which is the conditioning information for zero123:
zero123/zero123/ldm/data/simple.py
Lines 257 to 272 in f70ea8c
from zero123.
If you want to generate a novel view based on a sampled target viewpoint described by an RT matrix, assuming your input viewpoint is cond_RT, you can sample a target viewpoint target_RT, and use the get_T function to get d_T which is the conditioning information for zero123:
zero123/zero123/ldm/data/simple.py
Lines 257 to 272 in f70ea8c
Thank you! I will try it. Can I write the above code in gradio_new.py to get the target Rt matric?
from zero123.
This function does (cond_RT, target_RT) -> d_T. Sounds like you want (cond_RT, d_T) -> target_RT. You will need to implement that yourself.
from zero123.
This function does (cond_RT, target_RT) -> d_T. Sounds like you want (cond_RT, d_T) -> target_RT. You will need to implement that yourself.
cond_RT means input image RT, target_RT means synthesize view RT, what's mean d_T?
from zero123.
It means the relative transformation from cond_RT to target_RT, which is used as input to the diffusion model:
Lines 81 to 82 in f70ea8c
I suggest you try to understand the paper/code before raising an issue here. Closing the ticket for now.
from zero123.
Related Issues (20)
- Can export the mesh with texture
- getting some cv2 related error HOT 1
- Cant figure out where to put the model
- sd basemodel HOT 1
- UserWarning: ModelCheckpoint(monitor=βval/loss_simple_emaβ) not found in the returned metrics HOT 1
- how to create_carvekit_interface without the online access to huggingface?
- Partial dataset HOT 7
- RuntimeError: DataLoader worker (pid(s) 3360242, 3361527, 3362405) exited unexpectedly HOT 4
- test/benchmark on non-object data
- Can we continue training on the basis of 105000.ckpt? HOT 1
- Run time error `python gradio_new.py` HOT 2
- demo is down
- Partial Objaverse dataset for demo HOT 3
- batch size issue when trianing custom dataset HOT 1
- offline rendering or online rendering
- How to change the view of the input picture
- A question
- Zero123-xl download link is fail. HOT 1
- How to render custom data
- Can I finish all the tasks on a 4090?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zero123.