Why is reading a small number of specific columns with pandas read_csv usecols so slow?

user_13340814 0
Viewed 393

I'm trying to read in 14 columns of data from a csv with ~300 rows and ~3 million columns. To my understanding, the point of the usecols parameter in pandas.read_csv is that I can just read in the columns I want to save time and memory when I don't need all the columns.

However, reading in 14 columns with 300 rows takes almost a minute, regardless of whether or not I specify data types for all the columns and index (done using timeit):

Load time with dtypes: 55.114283496979624
Load time without dtypes: 54.37552756909281

I would expect reading a dataframe with shape (300,14) to be a lot faster than a minute. Is the slowness because pandas has to read all 3 million headers before deciding what columns to read in? How can I make this faster?

pandas python performance
1 Answers

Since the data is row-oriented; it will end up reading each line to get the columns it needs (unless the authors of pandas don't read a full line at a time, and somehow stop reading a line when you get the columns you need).

If you read it once, then .to_parquet("data.parquet"), it will be easier for you to read next time with .read_parquet("data.parquet", columns=[...]), this is because parquet orders the data in a column-oriented way. Even if you don't ask for specific columns, it might still be faster.

This helps in compressing the data very well, and it also helps in reading only the columns you need (or performing operations on them) easier.

To use it, you might need to pip install pyarrow.

user_70438050
Related