how to read multiple text files into a dataframe in pyspark

user_13665204 0
Viewed 3592

i have a few txt files in a directory(i have only the path and not the names of the files) that contain json data,and i need to read all of them into a dataframe.

i tried this:

df=sc.wholeTextFiles("path/*")

but i cant even display the data and my main goal is to preform queries in diffrent ways on the data.

rdd pyspark databricks dataframe sql
1 Answers

Instead of wholeTextFiles(gives key, value pair having key as filename and data as value),

Try with read.json and give your directory name spark will read all the files in the directory into dataframe.

df=spark.read.json("<directorty_path>/*")
df.show()

From docs:

wholeTextFiles(path, minPartitions=None, use_unicode=True)

Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI. Each file is read as a single record and returned in a key-value pair, where the key is the path of each file, the value is the content of each file.

Note: Small files are preferred, as each file will be loaded fully in memory.

user_76326950
Related