Loading data to BigQuery: is it possible to combine auto-detected schema with modifications?

Viewed 139

I am loading data from local csv to Big Query - Loading data to BQ.

I am using auto detect schema but it detects based only on first 100 rows: Auto detect schema samples up to the first 100 rows so if the column contains all integer for up to the first 100 rows then the data type will be integer

So if you have for example 100 integers and then in row 101 float - it will detect wrongly.

Is there a way, to combine - keep autodetected schema for every correctly detected column, and correct the schema on the columns it detected wrongly?

for example in python something like this:

def create_job_config(table_name):
    schema = None
    if "sth" in table_name:
        schema = [
            bigquery.SchemaField("client_id", "INT64"),
            bigquery.SchemaField("shop_id", "INT64")
        ]
    job_config = bigquery.LoadJobConfig(
        autodetect=True,
        schema=schema,
        skip_leading_rows=1,
        allow_quoted_newlines=True,
        # The source format defaults to CSV, so the line below is optional.
        source_format=bigquery.SourceFormat.CSV,
        field_delimiter=";"
    )
    return job_config

so something like: autoschema BQ detected corrected by list of corrections.

0 Answers
Related