Quickest ways to read large files with varying number columns in Python

user_4241483 0
Viewed 75

I have a data set of float numbers in a file data.txt, which contains a variable number of columns per row. For example:

    3.0 2.5
    1.1 30.2 11.5
    5.0 6.2
    12.2 70.2 14.7 3.2 1.1

In order to read it, I can easily use a combination of fopen and fscanf in Matlab. The last one reads and converts the data into an array in column order, like this one:

    array = [3.0 2.5 1.1 30.2 11.5 5.0 6.2 12.2 70.2 14.7 3.2 1.1]'

I want to translate my Matlab code into Python. But since there is no Python built-in function that replaces the Matlab's fscanf, I wrote the following Python code that reads and reshapes the data in the same way as described:

    from numpy import *

    data = []
    with open('data.txt') as file:
       for line in file:
          cline = line.split()
          data = data + cline

    data = array(data)

This works, but some of my data sets can have up to 200,000 rows, and the Python code that I've shown is very slow for the reading of large data sets (about 10 minutes). On the other hand, Matlab's fscanf does the job in just a couple of seconds, or less. So, is there any faster (optimized) way than my code to do this in Python?

I would really appreciate any suggestion.

pandas numpy file python
3 Answers

After a few thousand rows, this is doing tons of extra work:

    data = data + cline

Just data.extend(cline). (Or .append(), if you want to know which numbers appeared together on a line.)

Consider storing doubles instead of text:

    data.extend([float(c) for c in line.split()])
user_84311110

numpy.loadtxt would have been perfect here doesn't apply here because the number of columns change.

You want a flat list, you could speed it up a bit by using a list comprehension:

from numpy import *
with open("file.txt") as f:
    data = array([float(x) for l in f for x in l.split()])

(Now I'm pretty sure it will be much faster considering the mistake that JH pointed out in his answer: data = data + line creates a new list each time: quadratic complexity. You avoid that with the list comprehesion)

user_64515730

Pandas is much better/faster at handling ragged columns than numpy is, and should be faster than a vanilla python implementation with a loop.

Use read_csv, followed by stack, and then access the values attribute to return a numpy array.

max_per_row = 10 # set this to the max possible number of elements in a row

vals = pd.read_csv(buf, header=None, names=range(max_per_row),
                             delim_whitespace=True).stack().values

print(vals)
array([  3. ,   2.5,   1.1,  30.2,  11.5,   5. ,   6.2,  12.2,  70.2,
        14.7,   3.2,   1.1])
user_49090870
Related