Split data into multiple files: how to handle (unknown number of) multiple connections

user_3375672 0
Viewed 226

I want to split a (in real life: huge) file into multiple files specified by, say, the second column in data. I.e. in the example below I need files 431.csv and rr1.csv. My main idea was to open new connections to write if not already open - a record of open connections is in the dict files_dict, and then iterate through this and close in the end.

I am stuck in how to refer to these connections line by line.

In real life the number and value of these file names (second column) is not known beforehand.

Found some inspiration here:

write multiple files at a time

python inserting variable string as file name

How can I split a text file into multiple text files using python?

Content of toy data in data_in:

123,431,t
43,rr1,3
13,rr1,43
123,rr1,4

My naive pseudo-code as of now:

files_dict = dict() #dict of file names

with open(data_in) as fi:
    for line in fi:
        x = line.split(',')[1]

        if x not in files_dict:
            fo = x + '.csv'
            files_dict[x] = fo

            '''
            open files_dict[x]
            write line to files_dict[x]

            '''
    else:
        '''
        write line to files_dict[x]
        '''

for fo in files_dict.fos:
    fo.close()
file-io python split
3 Answers

You do have the right idea, but you should store the file objects rather than the file names in the dict, and you don't need an else block (which should've been aligned with if rather than for):

files_dict = {}

with open(data_in) as fi:
    for line in fi:
        x = line.split(',')[1]
        if x not in files_dict:
            files_dict[x] = open(x + '.csv', 'w')
        files_dict[x].write(line)

for file in files_dict.values():
    file.close()
user_68909120

Put the file objects themselves into the dictionary, not the filenames.

files_dict = {}

with open(data_in) as fi:
    for line in fi:
        x = line.split(',')[1]

        if x not in files_dict:
            fo = open(x + '.csv', "w")
            files_dict[x] = fo
        else:
            fo = files_dict[x]

        fo.write(x)

for fo in files_dict.values():
    fo.close()
user_14918950

You can also use pandas for your large csv as it handles it nicely, then just iterate through pandas column:

df = pd.read_csv('fun.txt', header=None)

string = "tester string"

for row in df[1]:
    fo = row + '.csv'
    f = open(fo, 'a')
    f.write(string+'\n')
    f.close()

output is 2 files, 431.csv and rr1.csv. Contents of 431.csv:

tester string

contents of rr1.csv:

tester string
tester string
tester string

It will append any added info to duplicate files, I feel like this is the desired behavior based on your pseudocode. This is a good solution because it will open and close your files as it loops through the column. That way you don't have 50 files open at the same time, which could cause troubles for your os.

user_84682640
Related