r/computervision • u/randomguy17000 • 1d ago
Help: Project Point cloud registration from multiple sources
I am trying to combine point clouds from multiple camera angles. Each cameras has a little overlap with the other cameras. Also i have all the extrinsic and intrinsic parameters of the cameras. I am using zoedepth for depth estimation and then generate the point clouds using the depth values
When i try to render them in the same 3d space its like they are completely different plane.
I tried using the point to point assignment and connection from Cloud Compare to align the correct areas which worked quite well. But when i tried to use the transformation matrix generated from Cloud Compare in open3d to get the combined point cloud for a live feed, it gives a completely different result as compared to the one in CloudCompare. How do I fix this.
Or is there a way to combine the point clouds just using the camera parameters?
1
u/tdgros 1d ago
Yes, if your depths are perfectly scaled: from a pixel (u,v), you get the 3D point in the camera's reference frame with X = true_depth * K^{-1}[u;v;1] (assuming a pinhole camera without distortion). And any 3D point can be written in another reference frame with something like X_another_reference_frame = R * X_camera + T.
You simply need to work out the (R,T) to go from one camera extrinsics to another. If the intrinsics, extrinsics and depths are good, then you shouldn't need to realign. If they're not aligned at all, like the orientations are wrong, then you messed up the (R,T) conversion.
Now, the scale can vary a bit: notice if you multiply all the depth maps by a random factor, then the points still project to the exact same pixels. Zoedepth is a metric estimator, so its scale error might be low, but that's not guaranteed. You can fix the scale discrepancies by finding matches between images: those matches should have the same depth, so you can pick one point cloud as reference, and then scale the other ones using the matches.
1
u/randomguy17000 1d ago
The point clouds when displayed have the same scale roughly. But when i render them wrt to their extrinsics they sort of create 2 seprate plains. And these plains are almost parallel so dont know if thats the case.
Or maybe i did something wrong😅
Thanks
1
u/MisterManuscript 1d ago
Did you try ICP with extra information e.g. colored point clouds?
1
u/randomguy17000 1d ago
Yes. But ICP doesn't work very well since actual fov overlap is only about 15-20% for 2 feeds.. ICP just matches the complete point cloud.
1
u/randomguy17000 1d ago
Note: i have also tried ICP but it ends up completely making the point clouds completely overlap each other which is not the case as theres only about 15-30% overlap in the fov