Fastest way to horizontally merge CSVs with different delimiters and columns in Python?

user_18854839 0
Viewed 48

I have one CSV (file1.csv) with odd data and delimiters that looks like this:

[Header]
Info1,Info2
Foo1,Foo2
Bar1,Bar2
[Data]
Info3,Info4
Foo3,Foo4

I am also able to convert it to a Pandas DF.

I have a second file (file2.csv) that looks like a traditional CSV:

Sample,ID,Project_Name,Description
Sample1,Example_ID_1,ProjectFoo,N/A
Sample2,Example_ID_2,ProjectFoo,N/A

I want to horizontally merge these CSVs (or Pandas DF then to CSV) into one output CSV that would just look like this:

[Header]
Info1,Info2
Foo1,Foo2
Bar1,Bar2
[Data]
Info3,Info4
Foo3,Foo4

Sample,ID,Project_Name,Description
Sample1,Example_ID_1,ProjectFoo,N/A
Sample2,Example_ID_2,ProjectFoo,N/A

I am having trouble merging these two using pd.concat or pd.merge because of the differing columns and delimiters and preserving the output CSV structure I want.

I know in bash I could probably just cat the two files but is there an efficient way to do this in Python? Currently, I make a subprocess call in shell to cat the two files in my Python script which I am not a fan of.

Any ideas?

pandas concatenation csv python
1 Answers

As you are not doing any processing on the data, just a simple concatenate, you could just read each file and write it to a single output file:

with open('output.csv', 'w') as f_output:
    for fn in ['file1.csv', 'file2.csv']:
        with open(fn) as f_input:
            f_output.write(f_input.read() + '\n')
user_49857330
Related