Create DataFrame columns dynamically from meta data

user_2265546 0
Viewed 123

I am new to Python and Databricks.

I have data in the text file without column headers.

'|' is the column value delimiter in this text file

I do have the details about the column names and their order in a separate excel file.

I have to create DataFrame with columns details which I do have then I have to import the data rows in it

Please suggest the best way to create DataFrame columns where should I maintain this columns details (Names, Type, Order)

Sample data in the text file:-

C1|CL1|23|dsfr|54|hjhg|nnf|hb
C1|CL2|23|dsfe|344|er|nnf||fg
C1|CL3|23|dsef|364|e|nnf|fg
C2|CL1a|2783|d56sfr|54|hjhg|nbvc|hb
C2|CL2a|253|dsf56e|344|er|hjhgf||fyyg
C2|CL3a|2673|dse56f|364|e|tre|ttt|yy

Header Details:-

Col1(string)
Col2(string)
Col3(number)
..
..

Please note that in reality, I do have more than 300 columns with string, number, DateTime, bool types

pandas azure-databricks databricks dataframe python-3.x
1 Answers

headers_df = pd.read_csv('path to headers csv') headers = headers_df.values.tolist()

Assign columns name

pd.read_csv('path/to/file/without_header.csv',sep='|',names=headers)

hope this answer your question

user_111525510
Related