Could not determine schema for table for uploading to bigquery from local using python

user_13803658 0
Viewed 1608

I'm trying to upload files locally into bigquery using python. Whenever i run this i get an error

ValueError: Could not determine schema for table 
Table(TableReference(DatasetReference('database-150318', 'healthanalytics'), 'pres_kmd'))'. Call client.get_table() or pass in a list of schema fields to the selected_fields argument.


client = bigquery.Client(project="database-150318")
job_config = bigquery.LoadJobConfig(autodetect=True)
table_ref = client.dataset('healthanalytics').table('pres_kmd')
table = client.get_table(table_ref)
#table = dataset.table("test_table")
    
deidrows = []
for filename in glob.glob('/Users/janedoe/kmd/health/*dat.gz'):
    with gzip.open(filename) as f:
        for line in f:
            #line = line.decode().strip().split('|')
            deidrows.append(line)
        client.insert_rows(table, deidrows)
        pdb.set_trace()

Can someone help with why? I already thought that if i put autodetect in there it would assume.
Thanks in advance!

google-bigquery uploading schema
1 Answers

You can try this example:

import csv
client = bigquery.Client()
table_ref = client.dataset('bq_poc').table('new_emp')
table = client.get_table(table_ref)

filename = "data.csv"
with open(filename) as f:
    for line in f:
        reader = csv.reader(f, skipinitialspace=True)
        rows = [[int(row[0]), str(row[1]), int(row[2])] for row in reader]
    client.insert_rows(table, rows)

Note:

  1. job_config is not utilized and it can be removed
  2. Data needs to be converted into specific format (mentioned as rows)
user_140485790
Related