I have written a load job in python using google colab for developing purposes but every time I run the code it loads the index into the bigquery table. However, when I run it on cloud fucntions the same code it does not load the index column. Index is the default index column pandas creates.
My code is as follow:
import pandas as pd
from google.cloud import bigquery
import time
from google.cloud import storage
import re
import os
from datetime import datetime, date, timezone
from datetime import date
from dateutil import tz
import numpy as np
job_config = bigquery.LoadJobConfig(
schema=[bigquery.SchemaField("fecha", bigquery.enums.SqlTypeNames.DATE)],
write_disposition="WRITE_TRUNCATE"
,create_disposition = "CREATE_IF_NEEDED"
,time_partitioning = bigquery.table.TimePartitioning(field="fecha")
#,schema_update_options = 'ALLOW_FIELD_ADDITION'
)
client = bigquery.Client()
job = client.load_table_from_dataframe(df, table_id,job_config=job_config)
My requirements.txt in cloud functions includes the following libraries
- pandas
- fsspec
- gcsfs
- google-cloud-bigquery
- pyarrow
- google-cloud-storage
- openpyxl