How to correctly fetch all genomic Coronavirus sequences along with their metadata using C3.ai Data Lake in Python?

My goal is to use Python and its requests library to extract sequence data from the C3.ai Data Lake using its fetch API. Using the resulting JSON, I plan on using the pandas library to create a dataframe that stores a bunch of sequence meta-data from the BiologicalAsset objects, along with columns for the actual sequence, its length, and its sequenceType fields, which come from the Sequence object. This data is printed and is then moved to R for further viewing of the results. However, I am having trouble getting the data that I want from C3.ai and I am not sure why.

In the code below, I specify that I am including the Biological Asset as well as the sequence child object, the length, and the sequence type. This is all working correctly because I am receiving a dataframe with all of the correct columns after I flatten the JSON with json_normalize(). I use the "filter": "sequence.length >= 28900", to filter out any sequences that are too short to be a complete genome. The problem is that I am only getting 105 results the fit this criteria and I know this is false because the NCBI Coronavirus data hub has almost 3000 sequences that it flags as complete. Futhermore, filtering for nucleotide sequences that have valid sequences in the child object using "filter": "assetType == 'nucleotide sequence' && exists(sequence.sequence)", yields the same result. Using "filter": "exists(sequence.sequence)", shows that there are only 1314 sequences of any type, including protein sequences, that actually have data in their child sequence object. Running these same searches using the sequence fetch API "https://api.c3.ai/covid/api/1/sequence/fetch" and the appropriate length and existence filters has given me the same numbers.

Browsing a table I made using an sequence fetch I made to retrieve all possible sequences by using no filter, I found that the sequence objects themselves existed for probably all of the sequences I need, but that the actual sequence fields themselves are missing. Is this an issue with my approach or with the Data Lake?

Thanks for your help!

Link to NCBI COVID 19 hub

https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?VirusLineage_ss=Severe%20acute%20respiratory%20syndrome%20coronavirus%202%20(SARS-CoV-2),%20taxid:2697049&SeqType_s=Nucleotide&Completeness_s=complete

Biological asset fetch code

import requests
import pandas

def retrieve_data():
    has_more = True
    data = pandas.DataFrame
    url = "https://api.c3.ai/covid/api/1/biologicalasset/fetch"


    jsonRequest =   {
    "spec": {
        "include": "this, sequence.sequence, sequence.length, sequence.sequenceType",
        "filter": "sequence.length >= 28900", 
       "limit": -1
    }
  }


    headers = {
        'Accept': 'application/json',
        'Content-Type': 'application/json'
    }

    response = requests.post(url=url, json=jsonRequest, headers=headers)

    result = response.json()
    print(result["hasMore"], type(result["hasMore"]))  # checks if any left over sequences
    flatDict = pandas.json_normalize(result["objs"])  # flattens JSON
    return flatDict

print(retrieve_data())  # runs function
print(retrieve_data().columns)  # prints column names

Sequence fetch JSON with possible filters commented out.

jsonRequest = {
        "spec": {
            #"include": "this, parent",
            #"filter": "sequenceType == 'GenBank' && length > 28900",
            #"filter": "sequenceType == 'complete'",
            #"filter": "exists(sequence)",
            #"filter": "length > 28900",


            "limit": -1
        }
    }