Comments (25)
@racinmat Thanks for your work in depth post-processing. But it is too hard for people who have no idea about rendering to understand.
To simplify, we only have to use the following formula to convert the depth value gather in the game to real depth in meters.
f=10003.814,n=0.15(default),d_game is the z-buffer value,d_real is the real depth value in meters.
from gtavisionexport.
@wujiyoung depth is in NDC, so you need to recalculate it using the incerse of projection matrix.
I describe it in my master thesis where I inspected the GTA V visualization pipeline. https://dspace.cvut.cz/bitstream/handle/10467/76430/F3-DP-2018-Racinsky-Matej-diplomka.pdf?sequence=-1&isAllowed=y
See part 3.6.3 where I describe relation between NDC and camera space. Camera space is in meters, so after transferring it from NDC to camera space you will have it in meters.
It is more described in part 5.1, where I demonstrate projection of points from meters to NDC and backwards.
from gtavisionexport.
from gtavisionexport.
from gtavisionexport.
check out https://github.com/umautobots/gta-postprocessing for postprocessing code.
from gtavisionexport.
Hopefully this article series could help a little bit.
from gtavisionexport.
Thanks for reply. I quite struggle with reading the data, since every page of this multipage tiff uses different flags.
Also, I struggle with 2D bounding boxes from the PostgreSQL database. Are they up to date?
Since they have all 4 points from range [0-1], I thought it would be sufficient to multiply X coords by width and Y coords by height, but that does not look right and bounding boxes do not have position which seems right when displayed over a screenshot.
from gtavisionexport.
@barcharcraz I tried that, but unfortumatelly, it contains only semantic segmentation for cars, not for other objects. And if I am not mistaken, it completely lacks depth data.
I wanted to use ImageViewer you have as part of solution with managed plugin, but it does not seem to be working.
from gtavisionexport.
from gtavisionexport.
The postprocessing code would be really great.
I checked the bounding box, but it is stored as box (native postgres structure), not box2d (postgis extension). And in the query building here coordinates are but there, not offset and extent, if I understand it correctly.
from gtavisionexport.
from gtavisionexport.
Oh, my bad with bounding boxes.
I was confused, because in the C# code, there was new NpgsqlBox(detection.BBox.Max.Y, detection.BBox.Max.X, detection.BBox.Min.Y, detection.BBox.Min.X)
but C# persists it to PostgreSQL in form of (MaxX, MaxY, MinX. MinY), which confused me. Now I can display them correctly.
But the method you proposed in your paper uses much better bounding boxes refinement. I was little bit disapointed that I could not find this post processing code, because you did really good job in refining data from both depth and stencil data.
You were right about the coarseness of these native bounding boxs. Really looking forward if you will decide to upload the post processing code. We wanted to use this repository at our university to replicate your research, and to prepare of our own dataset for some other tasks in field of machine learning.
from gtavisionexport.
@JiamingSuen. Hi, Have you figured out to decode the true depth value from the depth buffer?
from gtavisionexport.
@TommyAnqi As the author mentioned, the depth is already linearized. So no decoding is needed.
from gtavisionexport.
@JiamingSuen I want to access the real depth value with specific metric, such as meter, what should I do?
from gtavisionexport.
Well it's in "meters". Do be a little careful since while things like cars and people should be the right size, things like road length and building distance may be a little distorted, just because it's a game.
from gtavisionexport.
@racinmat thank you so much, your master thesis is very useful to me. I have two questions more.
- We can get entity position (
x, y, z
) by native callENTITY::GET_ENTITY_COORDS
, but how can we get the value ofw
, which is needed in transformation from world coordinate to camera coordinate in part 3.6.2. - How can we get the distance value
l
,r
,t
,b
in part 5.1?
from gtavisionexport.
@wujiyoung you won't get w value, since it is the coordinate in homogeneous coordinates. So usual way to treat points in homogeneous coordinates is setting w
to 1, do all your calculations and then devide all points by their w
value, which normalizes them from homogeneous coordinates back to 3D.
I did not caluclate the l
, r
, t
, b
directly since I need them only as fractions in the projection matrix, but they can be calculated from the field of view and height/width ratio. The exact creation of projection matrix from field of view, width, height and near clip are in this function: https://github.com/racinmat/GTAVisionExport-postprocessing/blob/master/gta_math.py#L159
it is part of my repo where I perform various postprocessing of data gathered from GTA.
from gtavisionexport.
@racinmat I used your code to transform NDC into meters, but I don't think the result is correct.
I just adjusted the input size into W=1024 & H=768.
construct_proj_matrix(H=768, W=1024, fov=50.0, near_clip=1.5)
Are there other things i have to change?
from gtavisionexport.
are you sure the near_clip is correct? You need to obtain it from the camera parameters, the default near_clip is 0.15. I left 1.5 here, because I tried some things at time of writing the code, if you have incorrectly set near_clip it messes things a lot. Same goes for fov (GTA camera field of view), but I think 50 is default value.
from gtavisionexport.
Hi @racinmat , thanks for your sharing. According to your thesis, it seems that the transformation from camera coordinate to NDC is perspective projection. So, does it matter that P_{3,2} (in projection matrix) is 1 or -1. If not, may I use the standard perspective projection matrix (like the one provided by DirectX) to get the depth values in meters?
from gtavisionexport.
it matters, because in the RAGE (engine used by GTA V), in the camera view coordinate system, camera is heading in direction of negative Z, so positive Z values are behind camera, and negative are in front of camera. AFAIK -1 handles this negative Z coordinate. But that is just orientation, so if you use 1 instead of -1, it should work if you care only about depth in meters, and not the whole transformation into camera view.
from gtavisionexport.
thanks, @racinmat
I misunderstood 0.15 is another default number for near clip, just like your magic far clip. XD
Besides, I found a interesting stuff.
from the function :
var data = GTAData.DumpData(Game.GameTime + ".tiff", new List(wantedWeather));
you can get the projection matrix directly by calling data.ProjectionMatrix
and with your code, NDC can be transform into meters perfectly.
(data.ProjectionMatrix would be different whether there's sky(infinite depth) in the screen or not.
but only the matrix with sky can be sued for depth transformation perfectly.)
from gtavisionexport.
yes, you can use the projection matrix directly, but it's inaccurate.
If you look at how projection matrix is calculated, model_view_projection matrix and model_view matrix are obtained, and projection matrix is obtain by matrix multiplication of these. Because of calculation of inverse of model_view matrix and matrix multiplication, you face numerical instabilities and resulting projection matrix is inaccurate.
Constructing projection matrix from parameters avoids these numerical stability issues.
If you compare it, constructed matrix is slightly more precise than the obtained one from code.
from gtavisionexport.
@Yannnnnnnnnnnn Thank you for helping dummies like me 👍 ! Appreciate your comment! And thanks to the original genius @racinmat. <3.
@racinmat I hope one day i will be able to understand your work in depth & in depth.
from gtavisionexport.
Related Issues (20)
- what is inside constant buffer?
- 3D Bounding boxes
- HRESULT 0x80004002 HOT 1
- module machine type 'x64' conflicts with target machine type 'x86' HOT 2
- where can we get the intrinsic parameter of the camera?
- Possible to get road segmentation? HOT 5
- Stencil and depth data
- Managed plugin could not be loaded
- Only NativeUI managed to load HOT 9
- Version problem about managed branch HOT 1
- When I press the N key,The following error may occur in the WriteToTiff method of ImageUtils. Try and fail several times. Is there a better solution
- GTA V does not start (launch) with GTAVisionNative.asi HOT 1
- Can you release a compailed asi file?
- No need for "experimental" here. HOT 1
- How to export specular map HOT 1
- GTA internal camera intrinsic matrix
- Native part generates color.raw and depth.raw with only pixels and stencil.raw totaly black HOT 1
- how to change the camera into the Orthophoto projection ?
- Depth distorted HOT 1
- Dataset link does not work HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gtavisionexport.