Can you apply a manually defined schema when you try to read delta files in databricks?

user_14707150 0
Viewed 865

I have created a manual schema by using structtype etc.

I have a silver level 1 layer in ALDS containing raw delta files. I need to apply the manual schema to this silver level 1 layer which will then be written back as silver level 2. So I simply need to read in the SL1 data apply the manual schema and then write it to SL2

I get the following error when I try to read in the data from SL1:

java.lang.UnsupportedOperationException: DeltaDataSource source does not support user-specified schema.

Does anyone know how to proceed? Below the code I used to read in the delta files.

vbak = spark.read.format(read_format).schema(vbak_schema).load(load_path)

azure-data-lake delta-lake databricks schema
1 Answers
When we are deals Delta , handling this scenario is little tricky as like .

    # Generate a DataFrame of loans that we'll append to our Delta Lake table
loans = sql("""
            SELECT addr_state, CAST(rand(10)*count as bigint) AS count,
            CAST(rand(10) * 10000 * count AS double) AS amount
            FROM loan_by_state_delta
            """)

# Show original DataFrame's schema
original_loans.printSchema()
 
"""
root
  |-- addr_state: string (nullable = true)
  |-- count: integer (nullable = true)
"""
 
# Show new DataFrame's schema
loans.printSchema()
 
"""
root
  |-- addr_state: string (nullable = true)
  |-- count: integer (nullable = true)
  |-- amount: double (nullable = true) # new column
"""
 
# Attempt to append new DataFrame (with new column) to existing table
loans.write.format("delta") \
           .mode("append") \
           .save(DELTALAKE_PATH)

""" Returns:

A schema mismatch detected when writing to the Delta table.
 
To enable schema migration, please set:
'.option("mergeSchema", "true")\'
 
Table schema:
root
-- addr_state: string (nullable = true)
-- count: long (nullable = true)
 
 
Data schema:
root
-- addr_state: string (nullable = true)
-- count: long (nullable = true)
-- amount: double (nullable = true)
 
If Table ACLs are enabled, these options will be ignored. Please use the ALTER TABLE command for changing the schema.

"""
user_95990910
Related