spark read schema from separate file

user_11022372 0
Viewed 769

I have my data in HDFS and it's schema in MySQL. I'm able to fetch the schema to a DataFrame and it is as below :

col1,string
col2,date
col3,int
col4,string

How to read this schema and assign it to data while reading from HDFS?

I will be reading schema from MySql . It will be different for different datasets . I require a dynamic approach , where for any dataset I can fetch schema details from MySQL -> convert it into schema -> and then apply to dataset.

pyspark apache-spark-sql apache-spark
1 Answers

You can use the built-in pyspark function _parse_datatype_string:

from pyspark.sql.types import _parse_datatype_string

df = spark.createDataFrame([
  ["col1,string"],
  ["col3,int"],
  ["col3,int"]
], ["schema"])

str_schema = ",".join(map(lambda c: c["schema"].replace(",", ":") , df.collect()))
# col1:string,col3:int,col3:int

final_schema = _parse_datatype_string(str_schema)

# StructType(List(StructField(col1,StringType,true),StructField(col3,IntegerType,true),StructField(col3,IntegerType,true)))

_parse_datatype_string expects a DDL-formatted string i.e: col1:string, col2:int hence we need first to replace , with : then join all together seperated by comma. The function will return an instance of StructType which will be your final schema.

user_7503760
Related