How to read all csv files that begin with consonant?

user_17469932 0
Viewed 128
import os
for file in os.listdir("/content/drive/MyDrive/BigData2021/Lecture23/datasets"):
    if file.endswith(".csv"):
        print(os.path.join(file))

cities.csv
airports.csv
data_scientist_salaries.csv

I want to read this CSV file with spark which begins consonant without tell CSV filename. How to do that?

pyspark apache-spark-sql apache-spark
1 Answers

Using wildcard [b-df-hj-np-tv-z]*.csv in the path should do the job:

df = spark.read.csv("/your_directory/datasets/[b-df-hj-np-tv-z]*.csv")
user_13865510
Related