Compare column by column

In this method I would like to find die biggest distance between two (adjecent) points for each column in a cv::Mat. In the end the corresponding points (which have the biggest distance to each other) should be returned.

To achive this, I already researched a lot and now I stuck at this code snippet:

cv::Mat mat;
std::vector<cv::Point> pointVec, finalPointVec;
std::vector<float> allDist;
for (int i = 0; i < mat.rows; i++) {

    for (int j = 0; j < mat.cols; j++) {
        c = mat.col(j);

        if (c.at<Vec3b>(i, j)[0] == 0
            && c.at<Vec3b>(i, j)[1] == 0
            && c.at<Vec3b>(i, j)[2] == 255) {
            cv::Point diPoint(j, i);

            pointVec.push_back(diPoint);

            if (pointVec[j].x == pointVec[j + 1].x) {
                //std::cout << pointVec[j].y << "\n";

                float diffY = pointVec[j].y - pointVec[j + 1].y;
                float diffX = pointVec[j].x - pointVec[j + 1].x;
                float dist = sqrt((diffY * diffY) + (diffX * diffX));
                for (int d = 0; d < pointVec[j].x; d++) {

                    allDist.push_back(dist);
                }
        }
    }
}

So I already iterate through the cv::Mat and also calculate the distance. Now I would like to implement finding the biggest distance for each column. Here I'm asking for your help, how I could realize it. Although I thought if (pointVec[j].x == pointVec[j + 1].x) should be fine to find the same columns, but it seems to be the wrong implementation. Also - how may I return those points, which have the largest distance to each other?

Maybe for some me clarification, here an image, how it should look like (the circled points should be those, which has to be returned):

I'm happy about any answer!