Fail to separate points for closest pair of points problem

I'm trying to solve a classical algorithmic problem on Closest pair of points. To do this, I use an O(n log n) algorithm from CLRS. Let me remind you what it is:

For a set of points on a plane P we separate them with a vertical line A, such that subsets P_L and P_R have the same number of points from P (points might be on that line A). Then, we make an array of these points X (sorted by x-coordinate), and build arrays X_L, X_R that contain points from subsets P_L and P_R respectively. We do the same with an array of these points Y (sorted by y-coordinate) and build arrays Y_L, Y_R.

As I understand, the separation line A doesn't depend on the way we sort points on the plane, so X_L will have the same amount of points as Y_L, and X_R as Y_R. The only difference between, say, X_L and Y_R is the order of the points in them. So I make the partition according to my understanding and my program fails in some cases (I compare the output of this algorithm with a naive O(n^2) version).

Do I understand the partition procedure correctly or am I missing something?

The partition part of my code attached:

struct point {
    int x;
    int y;
}

double find_min_dist(vector<point>& xs, vector<point>& ys, int num_p) {
    int mid;
    point median;
    double min_left, min_right, min_d;

    //if there are less than 3 points, we find min_dist by bruteforce
    if (num_p <= 3) {
        return slow_closest(xs, num_p);
    }

    vector<point> y_left, y_right;
    vector<point> x_left, x_right;
    vector<point> y_trunc;
    
    //find a place for partition
    mid = num_p / 2;
    if (2 * mid == num_p) {
        median = xs[mid];
    }
    else {
        mid = mid + 1;
        median = xs[mid + 1];   
    }

    //fill in X_L and X_R arrays
    int y_li = 0;

    for (auto i = 0; i < mid; ++i) {
        x_left.push_back(xs[i]);
    }
    for (auto i = 0; i < num_p - mid; ++i) {
        x_right.push_back(xs[mid + i]);
    }

    //fill in Y_L and Y_R arrays
    for (auto i = 0; i < num_p; ++i) {
        if (y_li < mid) {
           y_left.push_back(ys[i]);
           y_li++;
        }
        else if (y_li == mid) {
            y_left.push_back(ys[i]);
            y_right.push_back(ys[i]);
            y_li++;
        }
        else {
            y_right.push_back(ys[i]);
        }
    }

//place for other parts of the code, which are irrelevant for my question