Generating a labeled image from 3D point cloud in opencv

I was trying to generate a labeled image from a a 3D point cloud. For this I had to merge 10 given frames of segmented images to a fused pointcloud. After that I applied the individual transformation matrix for each given frame and then applied the transformation matrix of the image plane on which I want to project the points. I am given with the camera intrinsics. What I have done so far is,

1. Generated a fused pointcloud by merging 10 frames.
2. After that, I am trying to project these 3D points to a given image. For this I am using the following formula to convert the 3D points to 2D image coordinates: u = (fu * x)/z + u0 and v = (fv * y)/z + v0
3. After that I am checking whether my u and v are within the range of the image size. The size of the image is 940 * 560 (940 rows and 560 columns)

   if(u >= 0 && u <= 959) && (v >= 0 && v <= 539)
   

4. If the condition is satisfied then I am copying the color values to this coordinate

But, I am not getting the expected result. I am sharing my code. I am not understanding what I have done wrong:

for (int k = 0; k < 10; k++) {
        pose_camera2world = LoadPose(GetFileName(data_dir, k) + ".transform.txt");
        RGB = cv::imread(GetFileName(data_dir, k) + ".label.png", cv::IMREAD_UNCHANGED);
        depth = cv::imread(GetFileName(data_dir, k) + ".depth.png", cv::IMREAD_UNCHANGED);
        int rows = RGB.size[0]; // 960
        int cols = RGB.size[1]; //540

        for (int i = 0; i < cols; i++) {
            for (int j = 0; j < rows; j++) {
                auto z = depth.at<ushort>(j, i);
                auto y = (j - intrinsics.cy) * z / intrinsics.fy;   // for cols
                auto x = (i - intrinsics.cx) * z / intrinsics.fx;   // for rows
                world_coord << x, y, z, 1;
                Eigen::Vector4f res = pose_frame_101 * (pose_camera2world * world_coord);
                point3d << res[0], res[1], res[2];
                pc.vertices.push_back(point3d);
                pc.colors.push_back(RGB.at<cv::Vec3b>(j, i));

                int u = (intrinsics.fx * res[0] / res[2])  + intrinsics.cx;
                int v = (intrinsics.fy * res[1] / res[2]) + intrinsics.cy;
                if ((u >= 0 && u <= 959) && (v >= 0 && v <= 539) && map.at<uchar>(u, v) <= res[2]) {
                    map.at<uchar>(u, v) = res[2];
                    labels_101.at<cv::Vec3b>(u, v)[0] = RGB.at<cv::Vec3b>(j, i)[0];
                    labels_101.at<cv::Vec3b>(u, v)[1] = RGB.at<cv::Vec3b>(j, i)[1];
                    labels_101.at<cv::Vec3b>(u, v)[2] = RGB.at<cv::Vec3b>(j, i)[2];
                }
            }
        }
    }

My output looks like this:My output

Whereas it should like like this: The desired output