how to extract x,y coordinates from OpenCV "cv2.keypoint" object?

OpenCV provides a function for this. You can run:

pts = cv2.KeyPoint_convert(kp)

Here is my take (runable code):

import cv2, os
import numpy as np
import matplotlib.pyplot as plt

# INITIALISATION
filename = os.path.join('foo', 'bar.jpg')
img0 = cv2.imread(filename) # original image
gray = cv2.cvtColor(img0, cv2.COLOR_BGR2GRAY) # convert to grayscale
sift = cv2.xfeatures2d.SIFT_create() # initialize SIFT 
f, (ax1, ax2) = plt.subplots(1, 2) # create subplots

# DETECT AND DRAW KEYPOINTS
# sift.detect() returns a list of keypoints
# keypoint is a standard class of opencv (not just SIFT-related)
kp = sift.detect(gray,None) # calculates SIFT points
img1=cv2.drawKeypoints(gray,kp, None) # mae new image with keypoints drawn
ax1.imshow(img1) # plot 

# RETREIVE KEYPOINTS COORDINATES AND DRAW MANUALLY
# Reade these and make numpy array
pts = np.asarray([[p.pt[0], p.pt[1]] for p in kp])
cols = pts[:,0]
rows = pts[:,1]
ax2.imshow(cv2.cvtColor(img0, cv2.COLOR_BGR2RGB))
ax2.scatter(cols, rows)

plt.show()

I solved your problem like this.

kp,des = surf.detectAndCompute(img,None) 
pts = [p.pt for p in kp]

Now you get a list of x,y co-ordinates for all keypoints in your image.

to get the correct shape for optical flow input I used a combination of the above

kp = sift.detect(old_frame, None)
pts0 = cv2.KeyPoint.convert(kp).reshape(-1, 1, 2)