i want to know the best approach to extract data from teradata which is on mainframe server and data from hive on Hadoop platform and transform data using pyspark current approach :
- connecting to teradata using teradatasql and using pandas.read_sql to run query on teradata tables
- converting pandas dataframe to spark dataframe
issues:teradata query is taking very long time to execute()
while converting pandas dataframe to spark dataframe , it is throwing issue (no space left on device)
- using spark session , data from hive tables are stored in spark dataframe ( no issue)