batch tsv to csv script

user_15062007 0
Viewed 64

I'm pretty new to Python, I wrote this script that batch converts tsv files to csv. I keep getting an error message and spend hours trying to see what I did wrong. Any help on this will truly be appreciate it. Error code is "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte"

import os
import sys
import shutil
import pandas as pd
import argparse

def main():
    if len(sys.argv) == 1:
        files = [x for x in os.listdir('.') if x.endswith('.tsv')]
    else:
        files = [sys.argv[1]]
    
    for file in files:
        df = pd.read_csv(file, header=0, sep='\t', encoding='utf-8', quoting=3)
        new_filename = f'{file.replace(".tsv", "")}.csv'
        df.to_csv(new_filename, encoding='utf-8', index=False)
        print(f'Converted file: {new_filename}')

    print('Done!')

if __name__ == '__main__':
    main()
pandas dataframe csv python-3.x python
1 Answers

When the CSV is read into Pandas, it uses utf-8 encoding, however, there are other encoding formats that could have been used on the file

In this line of code:

df = pd.read_csv(file, header=0, sep='\t', encoding='utf-8', quoting=3)

Try setting encoding to a different format.

There are many different formats you can try, here is full list. I would recommend opening the file with notepad, or another text editor, and then save as a CSV with a utf-8 encoding.

user_126022080
Related