Passing a cursor to ndb on Google App Engine Python 3 leads to an error

user_975507 0
Viewed 216

Note: This is happening on my development server (MAC running Mojave)

I'm running Python 3 on Google App Engine (standard environment) and I have the code below

cursor = ndb.Cursor(urlsafe = next_page) if next_page else ndb.Cursor()
q = myObject.query(myObject.link == linkKey).order(-myObject.created)
resultsFuture = q.fetch_page_async(PAGE_SIZE,start_cursor=cursor)

If next_page is not None (which means cursor is not None), I get the following error

Traceback (most recent call last):
  File .../env/lib/python3.7/site-packages/google/cloud/ndb/_datastore_api.py", line 92, in rpc_call
    result = yield rpc
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.INVALID_ARGUMENT
    details = "Error parsing protocol message"
    debug_error_string = "{"created":"<CREATION_TIME>","description":"Error received from peer ipv6:<MY_IP_ADDRESS>","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Error parsing protocol message","grpc_status":3}"
>

Has anyone encountered this before and if so what was the solution?

If not, can anyone point me to a possible solution?

app-engine-ndb google-app-engine-python google-cloud-platform google-app-engine
2 Answers

https://github.com/googleapis/python-ndb/issues/471#issuecomment-649173225 says:

Hi, the cursor returned by fetch_page_async is already an instance of Cursor. The urlsafe argument to Cursor is a string that can be derived by calling Cursor.urlsafe() on a cursor instance. Since the cursor returned by fetch_page_async is already a Cursor instance, it can be passed directly to another call to fetch_page_async.

So instead you should do:

cursor = next_page if next_page else ndb.Cursor()
q = myObject.query(myObject.link == linkKey).order(-myObject.created)
resultsFuture = q.fetch_page_async(PAGE_SIZE,start_cursor=cursor)
user_3280360

I had a similar error in the past week and finally fixed it with some help, if you are still facing this issue kindly do the following:

cursor = ndb.Cursor(urlsafe = next_page.decode('utf-8')) if next_page else None
q = myObject.query(myObject.link == linkKey).order(-myObject.created)
resultsFuture = q.fetch_page_async(PAGE_SIZE,start_cursor=cursor)

This piece of code is really what fixes it:

next_page.decode('utf-8')

seeing as next_page is the urlsafe cursor_string you are sending and then later receiving from your frontend.

The problem seems to occur with Python3 when you create the Cursor from a urlsafe webstring, the issue is, in python2.7 the urlsafe() method in datastore_query.Cursor()used to be decoded one way or another to utf-8 but in python3 you have to do the conversion manually else what you send and later receive from the frontend will actually be a raw bytestring, mind you, this bytestring will convert smoothly to a Cursor object and even pass test but when sent as a _datastore_query.Cursor(), that's where the real problem arises as the Cursor created from that bytestring is wrong and leads to a status = StatusCode.INVALID_ARGUMENT error.

I hope this helps. Kindly let me know if you have any more question on this.

For some more context you can read on the error here: GRPC status code 3 for invalid arguments

Also you can try logging out the urlsafe (next_page) string you send and receive from your frontend to verify whether it is indeed not utf-8 decoded.

user_128345730
Related