iterate over a dataframe with save and restore each line

user_17465901 0
Viewed 171

I have a data frame like this

df1 = pd.DataFrame(
       {   
            "index":    ["December",  "December",  "November", "November"],
            "index1":   ["285",       "296",       "397",      "300"]})


     index     index1
0   December    285
1   December    296
2   November    397
3   November    300

Until now and for the purpose of my program I save one line every time and after that, I load/read it again and feed it into my program

so for example df1.head(1).to_csv("rules_fp2.csv") and after that df1 = pd.read_csv('rules_fp2.csv')

and so on for the following lines

However, I want to make it more automatic the procedure.

So I am trying to iterate over my dataframe and read each time one line from the dataframe and then save, load, and feed as before

Any ideas?

pandas dataframe python
2 Answers
import pandas as pd
import glob

for idx, row in df.iterrows():
    df.iloc[[idx]].to_csv(f'{dir_path}/row_num_{idx}.csv', index=False)

# read files
flist = glob.glob(f'{dir_path}/row_num_*.csv')
for fn in flist:
    df = pd.read_csv(fn)
    # do something with this df ...
user_72188710

As another option to looping is to apply a function for parallel processing. This will work for medium-sized dataframes with application of relatively simple/cheap functions (for more advanced cases, some optimization will be needed):

import pandas as pd

df1 = pd.DataFrame(
    {
        "index": ["December", "December", "November", "November"],
        "index1": ["285", "296", "397", "300"],
    }
)


def myfunc(row):
    print(row)
    # can add custom transform, save, read here
    return row


df1.apply(myfunc, axis=1)
user_106935960
Related