I have a Source Delta Table with all Strings and want to merge it into a Target Delta table with proper schema defined for integers, long etc.
root
|-- col1: string (nullable = true)
|-- col2: string (nullable = true)
|-- col3: string (nullable = true)
Target Schema example
root
|-- col1: integer (nullable = true)
|-- col2: short (nullable = true)
|-- col3: long (nullable = true)
Since spark does not allow us to read source table using user-given schema, I have to modify the source schema to existing schema. Is there any way to dynamically convert the schema of a dataframe to a target schema in PySpark which includes complex types (Nested objects and Lists as well)?