How to avoid `Unnamed` columns during read.csv using pandas?

user_10829044 0
Viewed 1638

I have 20-30 csv files to be read.

So, I try the below code

pat_dir = ['file*.csv']
files_grabbed = []
for files in pat_dir:
    files_grabbed.extend(glob.glob(files))
for f in files_grabbed:
    df = pd.read_csv(f, sep=",",low_memory=False)
    print(f)
    print(df.columns)

And printing them gives an output like below

file1.csv
Index(['Date', 'Code', 'Test', 'value', 'unit', 'TextualResults',
       'subject_id', 'class_id', 'Unnamed: 8',
       'Unnamed: 9', 'Unnamed: 10', 'Unnamed: 11', 'Unnamed: 12',
       'Unnamed: 13', 'Unnamed: 14', 'Unnamed: 15', 'Unnamed: 16',
       'Unnamed: 17', 'Unnamed: 18', 'Unnamed: 19', 'Unnamed: 20',
       'Unnamed: 21', 'Unnamed: 22', 'Unnamed: 23', 'Unnamed: 24',
       'Unnamed: 25', 'Unnamed: 26', 'Unnamed: 27', 'Unnamed: 28',
       'Unnamed: 29', 'Unnamed: 30', 'Unnamed: 31', 'Unnamed: 32',
       'Unnamed: 33', 'Unnamed: 34', 'Unnamed: 35', 'Unnamed: 36',
       'Unnamed: 37', 'Unnamed: 38', 'Unnamed: 39', 'Unnamed: 40',
       'Unnamed: 41', 'Unnamed: 42', 'Unnamed: 43', 'Unnamed: 44',
       'Unnamed: 45', 'Unnamed: 46', 'Unnamed: 47', 'Unnamed: 48',
       'Unnamed: 49', 'Unnamed: 50'],

While I can avoid the unnamed columns after read.csv using below code

df = df.loc[:, ~df.columns.str.contains('^Unnamed')]

How can I avoid reading those unnamed column during read.csv operation?

Please note that I don't know the column names before hand. So, I can't define column names to read.csv. because each file can have different column names

So, is there any way to drop them during read.csv operation because I have 30 files, and this causes issues during glob operation?

pandas glob dataframe csv python
2 Answers

How to remove Unnamed columns during reading a csv file?

Pandas read_csv method accepts an optional keyword argument called usecols which is used select a subset of columns from a csv file. The interesting thing about this argument is that it can accept a callable function, this callable function is then evaluated against the column names and return only the column names where the callable function evaluates to True.

Here is how you can pass the callable function in your example to prevent the Unnamed columns from being read in first place.

for file in files_grabbed:
    df = pd.read_csv(file, low_memory=False, 
                     usecols=lambda c: not c.startswith('Unnamed:'))
user_128331660

pd.read_csv('file.csv', index_col=0)

user_172304730
Related