I have deployed a code to read data from a drive folder. I store the last called pagetoken in SQL and run it in a loop to retrieve data for all items in the folder(>250k). The loop works in 2 ways, either it starts from the beginning and calls the nextPageToken iteratively or it uses the last nextPageToken value saved in my sql table to run the same loop.
When I use the token without calling from SQL, it works perfectly till all the data is read, but when called with SQL, it returns a bad request error. This process worked perfectly till I hadn't filtered my request by a modified time value.
def get_video_items(folder, service, token=None, begin=None, to=None):
# EXTRACT ALL ITEMS FROM SPECIFIED FOLDER
if not token:
size = PAGESIZE_2
response = service.files().list(q="'" + folder[0]['id'] + "' in parents and modifiedTime > '" + begin + "'and"
"modifiedTime < '" + to + "'",
orderBy='modifiedTime asc',
pageSize=size, fields='nextPageToken, files(id, name, webViewLink)').execute()
else:
size = PAGESIZE_2
response = service.files().list(q="'" + folder[0]['id'] + "' in parents and modifiedTime > '" + begin + "'and"
"modifiedTime < '" + to + "'",
orderBy='modifiedTime asc',
pageSize=size, fields='nextPageToken, files(id, name, webViewLink)',
pageToken=token).execute()
videos = response.get('files', [])
# RECORD PAGE TOKEN TO CALL NEXT PAGE
if len(videos) < size:
videos = None
else:
token = response.get('nextPageToken', None)
return videos, token
The error I receive is the following
raise HttpError(resp, content, uri=self.uri) googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/drive/v3/files?q=************+in+parents+and+modifiedTime+%3E+%272019-07-01T16%3A48%3A13.212522%27andmodifiedTime+%3C+%272022-07-01T10%3A38%3A13.212491%27&orderBy=modifiedTime+asc&pageSize=50&fields=nextPageToken%2C+files%28id%2C+name%2C+webViewLink%29&pageToken=~%21%21~AI9FV7Qu_zWbHy0zBnPqDegUffr0l2XrKvi_U_M8W17yWJ59skMILpiAR-0Oc_i-ZDxrj8Jx3AXBaMrqUh9fnrtxDcBEn4qKTHsdm4Y4_rnc6SJItgaXmOfx7HVjszfmG5z-bOHU-7flG0EIP1a2HIXq-GWFffu2ykGUaDACnteguHEjAe-tNEmPnRR1tHNmNEJACFgWnTnoCXb0w2M9kjA_VXGLwMtQ1Ar-Y9A6IJH3wQXxrIqVrIlWfJ-cEu0z6zGv3saM942Y4O0SKzLyTXk8-SWIozKH5pcFfeBTtn1z_OySgl-lHNIxRgu77lRy9wmgBhCAki8GHL0YjB_QXq7sodq7uwc9VPadokI7__0uoDCjAnvPkQ7gjH7VQWI80a1DadQ0GrZ3&alt=json returned "Invalid Value". Details: "[{'domain': 'global', 'reason': 'invalid', 'message': 'Invalid Value', 'locationType': 'parameter', 'location': 'pageToken'}]"
Is there something I'm doing wrong? Any help would be appreciated, thanks!