opencv-python pose estimation when dealing with non chessboard corners

In this Section

OpenCV shows a way to perform Pose Estimation with chessboard image

But when dealing with generic image , I can not just follow this tutorial So i use harris corner instead

    dst = cv.cornerHarris(gray, 2, 3, 0.04)
    coords = np.where((dst > 0.01 * dst.max()))
    imgpts = list(zip(coords[0], coords[1]))

then i code my program this

    corner = []
    for imgpt in imgpts:
        x = imgpt[0]
        y = imgpt[1]
        corner.append(np.mat([x, y]))

    corner = np.array(corner)

    if len(imgpts) > 0:
        corners2 = cv.cornerSubPix(gray, corner, (11, 11), (-1, -1), criteria)
        # Find the rotation and translation vectors.
        ret, rvecs, tvecs = cv.solvePnP(objp, corners2, mtx, dist)
        # project 3D points to image plane
        imgpts, jac = cv.projectPoints(axis, rvecs, tvecs, mtx, dist)

but i got this error

corners2 = cv.cornerSubPix(gray, corner, (11, 11), (-1, -1), criteria)
cv2.error: OpenCV(4.5.5) :-1: error: (-5:Bad argument) in function 'cornerSubPix'
> Overload resolution failed:
>  - Layout of the output array corners is incompatible with cv::Mat
>  - Expected Ptr<cv::UMat> for argument 'corners'

the chessboard corner format is like this:

[[[a,b],[c,d]...]]

It's all ndarray type,same as mine.

How can i do this right?

Edit1： according to opencv harris thanks to Micka 's help! I've obtained harris corners like this:

    dst = cv.cornerHarris(gray, 2, 3, 0.04)
    ret, dst = cv.threshold(dst, 0.01*dst.max(), 255, 0)
    dst = np.uint8(dst)
    ret, labels, stats, centroids = cv.connectedComponentsWithStats(dst)
    criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 100, 0.001)
    corners = cv.cornerSubPix(gray, np.float32(
        centroids), (5, 5), (-1, -1), criteria)

now I move on to

    # Find the rotation and translation vectors.
    ret, rvecs, tvecs = cv.solvePnP(objp, corners, mtx, dist)

Saddly when i assign objp to [[corner[0][0],corner[0][1],0]...]

Basically just add a third dimension value 0 to corner elements

    # 3D points : objp
    # objp = np.array([x, y, 0] for x, y in corner).T.reshape(-1, 2)
    objp = np.zeros((len(corner), 3), np.float32)
    for i in range(len(corner)):
        objp[i][0] = corner[i][0]
        objp[i][1] = corner[i][1]

    print(objp)

I got another error

cv2.error: OpenCV(4.5.5) /io/opencv/modules/calib3d/src/solvepnp.cpp:831: error: (-215:Assertion failed) ( (npoints >= 4) || (npoints == 3 && flags == SOLVEPNP_ITERATIVE && useExtrinsicGuess) || (npoints >= 3 && flags == SOLVEPNP_SQPNP) ) && npoints == std::max(ipoints.checkVector(2, CV_32F), ipoints.checkVector(2, CV_64F)) in function 'solvePnPGeneric'

Edit2 : upload data on Github