I am concatenating multiple html tables into a single dataframe. Unfortunately, one of the columns has a list of url links, so the Pandas 1.5 read_html(extract_links='all') doesn't work; it only grabbed the first link.
So I wrote a set of custom processing functions. collect_dataframes calls convert_html_to_dataframe from a list comprehension to create a list of dataframes. The final concatenated dataframe from collect_dataframes winds up with an empty row between each set of rows. The origin of the empty row is in the HTML table.
def collect_dataframes(self):
dfs = [self.convert_html_to_dataframe(file) for file in self.files]
return pd.concat(dfs, axis=0, ignore_index=True)
I need all of the completely empty rows dropped one way or the other. I'm currently dropping the row after a single table is converted to a dataframe in convert_html_to_dataframe. Is there a better way or place to do it at?
Also, could someone please explain why the second version of convert_html_to_dataframe doesn't work?
WORKS: drops the first row and returns dataframe
def convert_html_to_dataframe(self, file):
with open(file, 'r', encoding='utf-8') as f:
content = f.read()
df = pd.DataFrame(self.parse_html_table(content))
df.drop([0], inplace=True)
return df
DOES NOT WORK: returns an empty dataframe
def convert_html_to_dataframe(self, file):
with open(file, 'r', encoding='utf-8') as f:
content = f.read()
df = pd.DataFrame(self.parse_html_table(content))
return df.drop([0], inplace=True)