I'm writing some pretty simple code to parse a csv, apply a schema, save the file on gcp as a parquet, and then use bigquery to load many parquets in a single folder as the same table. So I read the csv in using pandas and apply the schema like this:
df = pd.read_csv(in_file, dtype=pd_schema, parse_dates=date_vars)
But there's an issue when a string column has one csv with all nulls and another with values. I'm not sure if this issue is happening on the parquet side or the bigquery side, but I get the error:
400 Error while reading table: temp, error message: Parquet column 'blah' has type BYTE_ARRAY which does not match the target cpp_type INT64. File: gs://blah/blah.parquet
It seems it gives an all-none object column the type INT64, but an object column with strings the type BYTE_ARRAY. Is there a way I can see if this issue is happening at the parquet level or the bigquery level? And is there a way I can be more specific in applying a schema? It seems the base issue is that pandas assigns the dtype Object when I specify str, is there maybe a pyarrow datatype I can apply that will be more specific? Thanks!