Is that possible to concat(merge) csv file without reading? (python, Jupyter)

Viewed 107

I have 2 csv files in my path. I want to concat or merge 2 files without reading. This is for data loading problem & merge error (huge data)

my folder/a.csv
my folder/b.csv

using code >> make a.csv + b.csv / not using pd.read_csv

3 Answers

One option to achieve this is using dask:

from dask.dataframe import read_csv, concat

df1 = read_csv('my folder/a.csv')
df2 = read_csv('my folder/b.csv')

final_df = concat([df1, df2])

final_df.to_csv('combined.csv', index=False, single_file=True)

This can be easily extended to the case of multiple csvs (assuming they have the same column structure):

from dask.dataframe import read_csv

final_df = read_csv('my folder/*.csv')
final_df.to_csv('combined.csv', index=False, single_file=True)

If you want to concat two files, you can write one file in another without reading the whole files. When you use the open function for CSV, you create a generator object and read it row by row. So, how it can look like:

import csv
with open('second_file.csv', mode='r', newline = '') as csvfile:
    with open('first_file.csv', mode='a') as f:
        reader = csv.reader(csvfile)
        writer = csv.writer(f)
        next(reader) #to skip csv header of second_file
        #you can even filter the second file without reading a whole and write filtered version. But if you don't need it, just use writer.writerows(reader)
        filt = filter(lambda x: int(x[2]) >= 2, reader) 
        writer.writerows(filt)
Related