efficiently reading digit matrix without delimiter in python

user_7357850 0
Viewed 51

I have a file which contains digit [0-9] matrix without delimiter with shape (N, M). N would be ~50k, M would be ~50k. A small version of matrix file will be for example, mat.txt

0012230012000
0012230002300
0012230004200

Right now I am using the following code, but I am not so satified with the speed.

def read_int_mat(path):
    """
    Read a matrix of integer with [0-9], and with no delimiter.
    """
    with open(path) as f:
        mat = np.array(
            [np.array([int(c) for c in line.strip()]) for line in f.readlines()],
            dtype=np.int8,
        )
    return mat

Edit: Here is a mini benchmark

import numpy as np
def read_int_mat(path):
    """
    Read a matrix of integer with [0-9], and with no delimiter.
    """
    with open(path) as f:
        mat = np.array(
            [np.array([int(c) for c in line.strip()]) for line in f.readlines()],
            dtype=np.int8,
        )
    return mat

%timeit read_int_mat("mat.txt")
%timeit np.genfromtxt("mat.txt", delimiter=1, dtype="int8")

print(read_int_mat("mat.txt"))
print(np.genfromtxt("mat.txt", delimiter=1, dtype="int8"))

Outputs are:

61.6 µs ± 1.32 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
327 µs ± 4.95 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
[[0 0 1 2 2 3 0 0 1 2 0 0 0]
 [0 0 1 2 2 3 0 0 0 2 3 0 0]
 [0 0 1 2 2 3 0 0 0 4 2 0 0]]
[[0 0 1 2 2 3 0 0 1 2 0 0 0]
 [0 0 1 2 2 3 0 0 0 2 3 0 0]
 [0 0 1 2 2 3 0 0 0 4 2 0 0]]

Is there any thing I could try to further speed it up. Would Cython help here? Many thanks in advance.

matrix python performance
1 Answers

You can use np.genfromtxt, for example:

The file (13 columns):

0012230012000
0012230002300
0012230004200

Then:

x = np.genfromtxt("file.txt", delimiter=1, dtype="int8")
print(x)

Prints:

[[0 0 1 2 2 3 0 0 1 2 0 0 0]
 [0 0 1 2 2 3 0 0 0 2 3 0 0]
 [0 0 1 2 2 3 0 0 0 4 2 0 0]]

EDIT: Version with np.fromiter and opening the file in binary mode:

def read_npfromiter(path):
    with open(path, "rb") as f:
        return np.array(
            [np.fromiter((chr(c) for c in l.strip()), dtype="int8") for l in f],
        )

Benchmark on file with shape (168, 9360):

from timeit import timeit


def read_int_mat(path):
    """
    Read a matrix of integer with [0-9], and with no delimiter.
    """
    with open(path, "r") as f:
        mat = np.array(
            [
                np.array([int(c) for c in line.strip()])
                for line in f.readlines()
            ],
            dtype=np.int8,
        )
    return mat


def read_npfromiter(path):
    with open(path, "rb") as f:
        return np.array(
            [np.fromiter((chr(c) for c in l.strip()), dtype="int8") for l in f],
        )


def f1(f):
    return np.genfromtxt(
        f, delimiter=1, dtype="int8", autostrip=False, encoding="ascii"
    )


def f2(f):
    return read_int_mat(f)


def f3(f):
    return read_npfromiter(f)


t1 = timeit(lambda: f1("file.txt"), number=1)
t2 = timeit(lambda: f2("file.txt"), number=1)
t3 = timeit(lambda: f3("file.txt"), number=1)

print(t1)
print(t2)
print(t3)

Result:

1.0680423599551432
0.28135157003998756
0.19099885696778074
user_100359850
Related