I have 2 csv files in my path. I want to concat or merge 2 files without reading. This is for data loading problem & merge error (huge data)
my folder/a.csv
my folder/b.csv
using code >> make a.csv + b.csv / not using pd.read_csv
I have 2 csv files in my path. I want to concat or merge 2 files without reading. This is for data loading problem & merge error (huge data)
my folder/a.csv
my folder/b.csv
using code >> make a.csv + b.csv / not using pd.read_csv
One option to achieve this is using dask:
from dask.dataframe import read_csv, concat
df1 = read_csv('my folder/a.csv')
df2 = read_csv('my folder/b.csv')
final_df = concat([df1, df2])
final_df.to_csv('combined.csv', index=False, single_file=True)
This can be easily extended to the case of multiple csvs (assuming they have the same column structure):
from dask.dataframe import read_csv
final_df = read_csv('my folder/*.csv')
final_df.to_csv('combined.csv', index=False, single_file=True)
Hey take a look at https://www.tensorflow.org/guide/data
You can work with https://www.tensorflow.org/guide/data#consuming_sets_of_files
Most of the operations in dataset don't load all the data as they are generators will load/preloaded data as it needs it.
You take and work with a subset of the datase with ds.take(number of items). It also allows to generate a pipeline including mapping functions. Once the pipeline is done you can iterate over the dataset to get all the data.
If you want to concat two files, you can write one file in another without reading the whole files. When you use the open function for CSV, you create a generator object and read it row by row. So, how it can look like:
import csv
with open('second_file.csv', mode='r', newline = '') as csvfile:
with open('first_file.csv', mode='a') as f:
reader = csv.reader(csvfile)
writer = csv.writer(f)
next(reader) #to skip csv header of second_file
#you can even filter the second file without reading a whole and write filtered version. But if you don't need it, just use writer.writerows(reader)
filt = filter(lambda x: int(x[2]) >= 2, reader)
writer.writerows(filt)