Getting the projection matrix of camera about unrealcv

[stavBodik]

@Vedhus I don't know, its should work maybe the syntax is different but you should find way to get the same using python.

Any way I figure out that in digital camera matrix (computer graphics like openGL/UnrealEngine) the pixel ratio equals to 1 because there is no physical distortion like in real life camera, this means that getting the intrinsic matrix is simple,no calibration operation needed :

Intrinsic Matrix:
K = [[f, 0, Cu],
[0, f, Cv],
[0, 0, 1 ]]

fx=fy=f=imageWidth /(2 * tan(CameraFOV * π / 360))

Cu=image horizontal center = imageWidth/2
Cv=image vertical center = imageHight/2

Enjoy .

[yilinliu77]

@Vedhus I don't know, its should work maybe the syntax is different but you should find way to get the same using python.

Any way I figure out that in digital camera matrix (computer graphics like openGL/UnrealEngine) the pixel ratio equals to 1 because there is no physical distortion like in real life camera, this means that getting the intrinsic matrix is simple,no calibration operation needed :

Intrinsic Matrix:
K = [[f, 0, Cu],
[0, f, Cv],
[0, 0, 1 ]]

fx=fy=f=imageWidth /(2 * tan(CameraFOV * π / 360))

Cu=image horizontal center = imageWidth/2
Cv=image vertical center = imageHight/2

Enjoy .

The focal length you calculate here is measured in pixel, but how can I get the focal length measured in real-world units(mm)? I can't find the sensor size of the default camera in the unreal engine.

Without the real focal length, how can I reproject the point into 3D world coordinates?

[dineshreddy91]

Hi @edz-o thanks for the reply
According to your formula the k matrix comes out to be

K = [320 0 320
0 320 240
0 0 1]
The depth to point cloud conversion using this intrinsic matrix is very distorted. Previously for opengl to camera matrix conversion i have used this https://strawlab.org/2011/11/05/augmented-reality-with-OpenGL. i wanted to access the OpenGL projection matrix to convert to the Hz intrinsic matrix.

[DonnEssime]

I have a quick note: the posted distance-to-image plane function does not seem to be correct; it actually computes the distance to the camera along the viewing direction (i.e. distance to image plane + focal depth).
The correct equation on line 8 should, imo, read

PlaneDepth = (PointDepth-1) / (1 + (DistanceFromCenter / f)**2)**(0.5)

However, for single-image reconstruction purposes it seems to work as it should 👍 I guess this is more of a semantic note, in that it does not compute the distance to the image plane, but rather to the Z plane in the camera's reference system.

[edz-o]

Hi,

@debadeepta
We guess the cause is that the depth image obtained by vget /camera/0/depth is the distance from the scene to the camera center, you may convert it to the distance to image plane using this function.

[DonnEssime]

For future reference, I got confused with the calculation/application of the location and rotation, because they do not follow the axes lay-out I was expecting. The translation axes for a non-rotated camera are given with the x axis into the scene, the y axis to the right and the z axis to the top (left-handed, for some reason?). I was expecting it to be z axis into the scene, x axis to the right and y axis to the bottom, as you do when projecting points onto an image plane.
By setting location = [location[1],-location[2],location[0]], the rest of my code ran as expected.

[edgarriba]

anyone faced: LogUnrealCV:Warning: error Argument Invalid
when calling vget /camera/0/proj_matrix ?

[qiuwch]

Hi, @dineshreddy91

This is an unfinished feature and I am working on it.
The camera matrix for FOV 90 is fixed, I need to ask my co-author for the exact value.

[dineshreddy91]

@qiuwch thanks for the reply. can you provide the projection matrix for FOV 90. i wanted to test the odometry and am stuck without the intrinsic matrix of the camera.

[edz-o]

Hi, @dineshreddy91

The focal length f = W / 2 / tan(FOV/2) = W/2, where W is the width of the image which is 640 px here. And the pixels are square.

[qiuwch]

@dineshreddy91 If this is not what you are looking for, then I need to spend one or two days to read Unreal Engine's code to find what is the correct answer.

[bhack]

https://docs.unrealengine.com/latest/INT/API/Runtime/Engine/Engine/ULocalPlayer/GetProjectionData/index.html

[debadeepta]

Using the K matrix above I am getting a really distorted point cloud as well. See attached image. It is a view into the room in RealisticRendering binary example. The depth image converted to point cloud using K above makes the scene 'cave-in' towards the center. I think we need the camera distortion model and coefficients to properly project. Is anyone else having the same experience?

[debadeepta]

Thank you! I didn't realize the depth image was in ray-distance form. Your function works great. Just a small nitpick: 'f' is undefined in your function. Ideally should be an argument to it.

Many thanks!

[bhack]

Could we add this to the plugin?

[qiuwch]

@bhack I am willing to finish the uncompleted vget /camera/[id]/proj_matrix. I am working on #60 and it is almost finished. This will be my next step. If anyone has already done this and can make a pull request, that would be best.

@debadeepta I think the distance to the image plane is a much natural definition for depth. The ray-distance should at least be clearly documented, this is our fault. I am thinking to provide both these two types of depth, but not sure how to call them, depth and ray_distance?

[qiuwch]

Hi, @DonnEssime

@edz-o is travelling these days, I will ask him to follow up this issue when he is back.

[edz-o]

Hi, @DonnEssime

You are right, this function computes the distance to the camera plane, i.e. the Z plane.

[jeffreykxiao]

@edgarriba as of the current version 0.3.10, the function vget /camera/0/proj_matrix is incomplete. It just contains a couple lines of commented out code and returns FExecStatus::InvalidArgument.

[stavBodik]

@dineshreddy91

CameraMatrix = IntrinsicMatrix * ExtrinsicMatrix

Where ExtrinsicMatrix = RotationMatrix|TranslationMatrix(aKa position)

For C++ users :

ProjectionMatrix projMatrix = client.simGetCameraInfo("0").proj_mat;

VectorMath::Matrix3x3f rotationMatrix = client.simGetCameraInfo("0").pose.orientation.toRotationMatrix();

VectorMath::Vector3f cameraPosition = client.simGetCameraInfo("0").pose.position;

[Vedhus]

@stavBodik is the projMatrix the IntrinsicMatrix?

[stavBodik]

@Vedhus No, The projection matrix is the camera matrix.

[Vedhus]

@stavBodik thanks, do you know if I can use the camera matrix directly with the cv2.reprojectImageTo3D?

[stavBodik]

@Vedhus "In computer vision a camera matrix or projection matrix is a matrix which describes the mapping of a pinhole camera from 3D points in the world to 2D points in an image. " I never tried but this matrix should help you to do the reverse operation if you know the depth/distance from the camera to each pixel.

[Vedhus]

@stavBodik Thanks, last question. Does this work with the python client as well?
proj = client.simGetCameraInfo("0").proj_mat

[Vedhus]

@stavBodik Thanks! Do you know why the projection matrix is 4x4 and not 3x4? Because the Intrisic matrix is 3x3 and extrinsic matrix is 3x4.

[stavBodik]

@Vedhus

Some calibration software provides a 4x4 matrix, which adds an extra row to preserve the z-coordinate. In this case, just drop the last row to get a 3x4 matrix.

http://ksimek.github.io/2012/08/14/decompose/

[cloudlakecho]

@whatseven, Are you able to get information from specific camera in Python wrapper? It looks like we are not able to attain information in Python environment.,..Here are others' experience.

[itwpub]

thanks

[Shromm-Gaind]

Is anyone else also getting this format of Projection Matrix? It seems wrong, I have no idea why I am getting this. I expected to get a matrix in this format:

ProjectionMatrix { 'matrix': [ [ 0.0,
1.0,
0.0,
0.0],
[ 0.0,
0.0,
-1.7777777910232544,
0.0],
[ 0.0,
0.0,
0.0,
10.0],
[ -1.0,
0.0,
0.0,
0.0]]}

[Nasr1359]

Is anyone else also getting this format of Projection Matrix? It seems wrong, I have no idea why I am getting this. I expected to get a matrix in this format: ProjectionMatrix { 'matrix': [ [ 0.0, 1.0, 0.0, 0.0], [ 0.0, 0.0, -1.7777777910232544, 0.0], [ 0.0, 0.0, 0.0, 10.0], [ -1.0, 0.0, 0.0, 0.0]]}

Hi I got the same Projection matrix when I tried to get the camera information of the AIRSIM simulator by using client.simGetCameraInfo("0") in python.