How do I get total row count of Pandas Dataframe when using iterator=True

user_4583366 0
Viewed 1088

I'm dealing with large files that doesn't fit in memory, as a result of that I'm using the iterator functionality of Pandas' Dataframe and processing single chunk each time.

pd.read_csv(csv_file_name, encoding='utf-8', chunksize=chunk_size, iterator=True,
                                            engine='c', error_bad_lines=False, low_memory=False)

While processing I'd like to print the number of processed rows and the percentage of processed rows out of the total amount of rows.

To get the total amount of rows in a Pandas Dataframe I'm using 

len(df.index)

But when trying to use it when using ierator I'm getting 

AttributeError: 'TextFileReader' object has no attribute 'index'

Any way of doing that? (while not going over each chunk)

pandas
1 Answers

Two possible work arounds I would use:

  1. Use the columns option and read the file in with just one column. It may be that is small enough you can read in one go, but if not iterate over that to count the number of rows.

  2. Use the linux command wc -l to count the number of lines. If you have a header you need to remove one. e.g.

wc_output = subprocess.run(['wc','-l', 'csv_file_name'])
# wc_output.stdout will be of format ` N_lines filename`
# subtract 1 to remove header
n_rows = int(wc_output.stdout.split()[0]) - 1
user_90722010
Related