I 've generated a huge (6G) txt file using a windows command line program (samtools.exe):
.\samtools.exe mpileup -O bamfile.bam > txtfile.tsv
The generated file is actually a table separated by tab. When I tried to use pandas.read_table to open it, it gives me:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
When I tried to print the first line of the file, it is like this:
ÿþAL645882 473 N 1 ^!c I 1
Everything is normal except the first character. If I read it use 'rb', indeed the first character is 0xff.
I really want this table to be read as a pandas DataFrame, the file is huge, is there anyway I can let python ignore the 0xff byte? Or simply delete the byte in the file?
Thanks in advance!