Fail to reconstuct objects with RGBD image

[jc-bao]

👋 Hi everyone. I am trying to use the reconstruction system to rebuild a cube from RGBD images. I collected 60 RGBD images from Pybullet by moving the camera around the cube. (the collected image can be found here. )
When I am trying to make fragments from the collected data following the tutorial, the script only returns JSON files without ply files. In the JSON file, it seems every transformation is an identity.
So my confusion is:

1. What might be the reason that the making fragmentation procedure fails?
2. Is the data I collected from the simulator correct? Or is this because I did not set up the parameters correctly?
3. Dose Open3D provide the protocol to integrate the initial measurement of camera position to assist the ICP?

Supplementary
The parameters I used for building fragmentation:

{
    "name": "Open3D reconstruction tutorial",
    "path_dataset": "data/cube/",
    "path_intrinsic": "bullet_cam.json",
    "max_depth": 3.0,
    "voxel_size": 0.05,
    "max_depth_diff": 0.07,
    "preference_loop_closure_odometry": 0.1,
    "preference_loop_closure_registration": 5.0,
    "tsdf_cubic_size": 3.0,
    "icp_method": "color",
    "global_registration": "ransac",
    "python_multi_threading": true,
    "n_frames_per_fragment": 15,
    "min_depth": 0.1,
    "max_depth": 3.0
}

Any information will be highly appreciated!

The scenario in that I collect data:

[yuecideng]

1. Hi, could you see any infomation printed in the console? Look at your provided camera json, it seems that it is illegal.

{
  "width": 1024, 
  "height": 1024,
  "fx": 1, 
  "fy": 0, 
  "cx": 1, 
  "cy": 0
}

The cx, cy should be about half of width and height. and fx and fy is abnormal...
3. Yes, you can use integrate_scene.py to integrate the data with given odometry.

[jc-bao]

Thank you so much for the timely reply. I have updated the camera JSON, but I still cannot generate a point cloud from the RGBD image.

The console shows [Open3D WARNING] PinholeCameraParameters read JSON failed: the wrong format. It seems to be caused by incorrect camera settings. I might have messed it with the camera's internal projection matrix.
The camera I used in the simulation has 90 deg of FOV, and its distance to the object center is 1 meter. So I use fx=fy=2/1024 and cx=cy=512. The generated JSON file is as follows,

{
	"height" : 1024,
	"intrinsic_matrix" : [ 0.001953125, 0.0, 0.0, 0.0, 0.001953125, 0.0, 512.0, 512.0, 1.0 ],
	"width" : 1024
}

This time the WARNING has gone, but I still cannot get the point cloud. The transformation in the generated graph file is still an identity.

Here is a snip of console output:

Fragment 003 / 003 :: integrate rgbd frame 48 (4 of 15).
[Open3D DEBUG] Validating PoseGraph - finished.
[Open3D DEBUG] [GlobalOptimizationLM] Optimizing PoseGraph having 15 nodes and 17 edges.
[Open3D DEBUG] Line process weight : 0.001960
[Open3D DEBUG] [Initial     ] residual : 0.000000e+00, lambda : 1.600000e-04
[Open3D DEBUG] Maximum coefficient of right term < 1.000000e-06
[Open3D DEBUG] [GlobalOptimizationLM] Optimizing PoseGraph having 15 nodes and 17 edges.
[Open3D DEBUG] Line process weight : 0.001960
[Open3D DEBUG] [Initial     ] residual : 0.000000e+00, lambda : 1.600000e-04
[Open3D DEBUG] Maximum coefficient of right term < 1.000000e-06
[Open3D DEBUG] CompensateReferencePoseGraphNode : reference : 0
[Open3D DEBUG] Validating PoseGraph - finished.
[Open3D DEBUG] [GlobalOptimizationLM] Optimizing PoseGraph having 15 nodes and 17 edges.
[Open3D DEBUG] Line process weight : 0.001960
[Open3D DEBUG] [Initial     ] residual : 0.000000e+00, lambda : 1.600000e-04
[Open3D DEBUG] Maximum coefficient of right term < 1.000000e-06
[Open3D DEBUG] [GlobalOptimizationLM] Optimizing PoseGraph having 15 nodes and 17 edges.

I am wondering where could go wrong. 🤔

[yuecideng]

Hi, your focal length is incorrect. Please see this tutorial.

[jc-bao]

Thanks for the advice! I have updated my JSON to

{
	"height" : 1024,
	"intrinsic_matrix" : [ 512.0, 0.0, 0.0, 0.0, 512.0, 0.0, 512.0, 512.0, 1.0 ],
	"width" : 1024
}

But unfortunately, the result is the same.

[yuecideng]

One possible reason is that you used incorrect depth_scale (this variable converts the depth value of depth image into real world depth in meter).

[jc-bao]

Oh yeah, my depth parameters might be wrong. In the simulation, my camera has a minimum depth distance of 0.2 and a maximum depth distance of 3.0. The rendered depth image was a value between [0,1).
To save the image with PIL, I use the following code:

    depth = Image.fromarray(np.uint8(depthImg*255), mode='L').convert('RGBA')
    depth.save(f'data/cube/depth/depth{i+1}.png')

I have updated depth_scale to 256/2.8=91.4. I have also changed the max_depth_diff from 0.07 to 0.1. But I still cannot make the fragmentation.
Could it be an issue with my depth image saving method?

[yuecideng]

I would suggest you to use uint16 to save depth image because the resolution would be more suitable for reconstruction usage.

In this case, your depth_scale is 65536 / 3. But I think this is not the critical problem for the fail of making fragment. I saw the depth image your provided (as follow). The backgraoud is write which means its value is 255, but I think it should be 0 because you don't want it, and the odometry computation will use the depth information, which might be an effect too.

[jc-bao]

Thank you so much for the advice! After reverting the depth image, I can successfully make the fragmentation.
So in Open3D, does a larger depth value indicate a smaller depth?

[jc-bao]

Another wired thing is the constructed point cloud is not a cube but a prismatic cone. In another word, the surface angle becomes less than 90 deg. Could it be caused by placing the camera too close to the cube or improper setting of the camera parameters?
The cube size is 0.4m*0.4m*0.4m, and the camera distance to it is 1m.

|

[yuecideng]

So in Open3D, does a larger depth value indicate a smaller depth?

No, Open3D works as the same with the most common libraries to deal woth depth image.

The backgraoud is write which means its value is 255, but I think it should be 0

What I mean is that, you should try to adjust your simulated camera in simulation to obtain the depth image with background value eqaul to 0. Or you can simply use the following code to change the current depths:

depth[depth == 255] = 0

[jc-bao]

Oh, thanks for the advice! After setting the background to zero rather than inverting the whole image, I can reconstruct the object now.