Need to drop completely empty rows from concatenated dataframe

Viewed 27

I am concatenating multiple html tables into a single dataframe. Unfortunately, one of the columns has a list of url links, so the Pandas 1.5 read_html(extract_links='all') doesn't work; it only grabbed the first link.

So I wrote a set of custom processing functions. collect_dataframes calls convert_html_to_dataframe from a list comprehension to create a list of dataframes. The final concatenated dataframe from collect_dataframes winds up with an empty row between each set of rows. The origin of the empty row is in the HTML table.

def collect_dataframes(self):
    dfs = [self.convert_html_to_dataframe(file) for file in self.files]
    return pd.concat(dfs, axis=0, ignore_index=True)

I need all of the completely empty rows dropped one way or the other. I'm currently dropping the row after a single table is converted to a dataframe in convert_html_to_dataframe. Is there a better way or place to do it at?

Also, could someone please explain why the second version of convert_html_to_dataframe doesn't work?

WORKS: drops the first row and returns dataframe

def convert_html_to_dataframe(self, file):
    with open(file, 'r', encoding='utf-8') as f:
        content = f.read()
    df = pd.DataFrame(self.parse_html_table(content))
    df.drop([0], inplace=True)
    return df

DOES NOT WORK: returns an empty dataframe

def convert_html_to_dataframe(self, file):
    with open(file, 'r', encoding='utf-8') as f:
        content = f.read()
    df = pd.DataFrame(self.parse_html_table(content))
    return df.drop([0], inplace=True)
1 Answers

Both responses worked on a table of concatenated dataframes, but I went with @Echo's response in the end since it was easier to remember later on: df.dropna(axis=0, how='all')

Related