Repeat the same process for all CSVs in a folder

user_17969786 0
Viewed 48

I have been trying to repeat a simple set of processes to about 50 CSVs from the same directory. I understand that I need to do a for-loop to repeat it, and the following is what I have done. However, despite getting no error message in my jupyter notebook, no changes has been made to my files. What have I done wrong?

I am beginner who just started learning Python.

import glob 
import os
import pandas as pd

path = "r'C:\3. Technical\3.4 GIS\1.1 Incoming\220107_Flow Data\test\WD_AM_PEAK\'"
all_files = glob.glob(os.path.join(path, "*.csv"))

for file in all_files:
    file_name = os.path.splitext(os.path.basename(file))[0]
    df = pd.read_csv(file_name)
    df.dropna(subset=['oo'])
    df.insert(3, "ODSegment", df['entrances'] + ' - ' + df['exits'])
    df['WD21AMoovph'] = df['oo']/100*df['estimatedCarsPerHour']
    df.rename(columns={"oo": "WD21AMoo", "estimatedCarsPerHour": "WD21AMvph"})
    df.to_csv(file_name, index=False, header=True)

Thank a lot.

pandas csv for-loop python
1 Answers

every time you operate on the dataframe it returns a new dataframe leaving the original untouched. at the end when you write df.to_csv(...) it is simply writing the original data back at that location.

try:


for file in all_files:
    file_name = os.path.splitext(os.path.basename(file))[0]
    df = pd.read_csv(file_name)
    df = df.dropna(subset=['oo'])
    df = df.insert(3, "ODSegment", df['entrances'] + ' - ' + df['exits'])
    df['WD21AMoovph'] = df['oo']/100*df['estimatedCarsPerHour']
    df = df.rename(columns={"oo": "WD21AMoo", "estimatedCarsPerHour": "WD21AMvph"})
    df.to_csv(file_name, index=False, header=True)
user_25702610
Related