pandas read_csv remove blank rows

user_4648472 0
Viewed 26878

I am reading in a CSV file as a DataFrame while defining each column's data type. This code gives an error if the CSV file has a blank row in it. How do I read the CSV without blank rows? 

dtype = {'material_id': object, 'location_id' : object, 'time_period_id' : int, 'demand' : int, 'sales_branch' : object, 'demand_type' : object }

df = pd.read_csv('./demand.csv', dtype = dtype)

I thought of one workaround of doing something like this but not sure if this is the efficient way:

df=pd.read_csv('demand.csv')
df=df.dropna()

and then redefining the column data types in the df.

Edit : Code - 

import pandas as pd
dtype1 = {'material_id': object, 'location_id' : object, 'time_period_id' : int, 'demand' : int, 'sales_branch' : object, 'demand_type' : object }
df = pd.read_csv('./demand.csv', dtype = dtype1)
df

Error - ValueError: Integer column has NA values in column 2

My CSV file's snapshot - enter image description here

blank-line pandas csv python
4 Answers

This worked for me.

def delete_empty_rows(file_path, new_file_path):
    data = pd.read_csv(file_path, skip_blank_lines=True)
    data.dropna(how="all", inplace=True)
    data.to_csv(new_file_path, header=True)
user_118026240

try smth like this:

data = pd.read_table(filenames,skip_blank_lines=True, a_filter=True)
user_27159480

solution could be :

data = pd.read_table(filenames,skip_blank_lines=True, na_filter=True)
user_87713380

I am not sure whether its efficient or not but it works. This code does not load nan values while reading a csv.

df=pd.read_csv('demand.csv').dropna()
user_105471040
Related