dataframe.py", in __getattr__ AttributeError: 'DataFrame' object has no attribute 'index'

Question

from pyspark.sql import SparkSession
from pyspark.sql.types import StringType
import pandas as pd
from google.cloud import bigquery
from google.cloud import storage
from pandas import DataFrame



from pyspark.sql import SparkSession
  
spark = SparkSession.builder \
  .appName('Optimize BigQuery Storage') \
  .config('spark.jars.packages', 'com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.24.2') \
  .getOrCreate()
  
df = spark.read.text('gs://test111/test/test.txt', lineSep=",")
  
#df.selectExpr("split(value, ',') as\
#Text_Data_In_Rows_Using_Text").show(4,False)

df.show()

sProjectID = 'prj-d-xxxx-ingest'
sTargetDataset = 'data'
sTargetTable = 'xxx'

client = bigquery.Client()
table_id = 'data.xxx'

# Since string columns use the "object" dtype, pass in a (partial) schema
# to ensure the correct BigQuery data type.
job_config = bigquery.LoadJobConfig(schema=[
    bigquery.SchemaField("Source_Code", "STRING"),
    bigquery.SchemaField("FLAG", "STRING")
])

job = client.load_table_from_dataframe(
    df, table_id, job_config=job_config
)

ERROR as below File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1643, in getattr AttributeError: 'DataFrame' object has no attribute 'index'

user_14733669 · Answer

Instead of using google.cloud.bigquery, you can use BigQuery connector with spark to write data to BQ. See BQ connector with spark document.

Assuming that your data is already correct on your df variable. You can apply this code to write the data to BQ:

gcs_bucket="your-gcs-bucket" #If not provided, it will create a temporary bucket for this

df.write.format("bigquery").option("table","dataset.your_table_here").option("temporaryGcsBucket", gcs_bucket).mode("append").save()
#.mode() can be either 'append' or 'overwrite'

I executed the code using spark-submit with Spark BigQuery jar:

spark-submit --jars gs://spark-lib/bigquery/spark-bigquery-latest.jar my_script.py

See logs after running the script:

dataframe.py", in getattr AttributeError: 'DataFrame' object has no attribute 'index'

1 Answers