Transposing files from columns to rows for multiple files

2Black_Cats 0
Viewed 24

I have approximately 200 files (plus more in the future) that I need to transpose data from columns into rows. I'm a microbiologist, so coding isn't my forte (have worked with Linux and R in the past). One of my computer science friends was trying to help me write code in Python, but I have never used it before today.

The files are in .lvm format, and I'm working on a Mac. Items with 2 stars on either side are paths that I've hidden to protect my privacy.

The for loop is where I've been getting the error, but I'm not sure if that's where my problem lies or if it's something else.

This is the Python code I've been working on:

import os
lvm_directory = "/Users/**path**"
output_file = "/Users/**path**/Transposed.lvm"
newFile = True
output_delim = "\t"
for filename in os.listdir(lvm_directory):
    header = []
    data = []
    f = open(lvm_directory + "/" + filename)
    for l in f:
        sl = l.split()
        if (newFile):
            header += [sl[1]]
    f. close()

This is the error message I've been getting and I can't figure out how to work through it:

  File "<pyshell#97>", line 5, in <module>
    for l in f:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 345: invalid continuation byte

The rest of the code after this error is as follows, but I haven't worked through it yet due to the above error:

        f = open(output_file, 'w')
        f.write(output_delim.join(header))
        newFile = False
    else:
        f = open(output_file, 'a')
    f.write("\n"+output_delim.join(data))
    f.close()
python
1 Answers

Looks like your files have a different encoding than the default utf-8 format. Probably ASCII. You'd use something like:

with open(lvm_directory + "/" + filename, encoding="ascii") as f:
    for l in f:
        # rest of your code here

^ It's generally more "pythonic" to use a with statement to handle resource management (i.e. opening and closing a file), hence the with approach demonstrated above. If your files aren't ASCII, see if any other encoding work. There are command-line tools like chardet that can help you identify the file's encoding.

Maurice Pasternak0
Related