Appending pandas Dataframe to each other inside loop

user_12319746 0
Viewed 181

I am getting a table on the page. The first page shows 100 rows and the next page the next 100 rows and so on. I have tried this, but this doesn't seem to work, It writes to only first table. What would be the correct way so that the next rows get keep getting added to the df.

tables = driver.find_elements_by_tag_name('table')
            table = tables[1].get_attribute('outerHTML')
            df_i = pd.read_html(table)
            df = pd.concat(df_i)
            while True:
                try:
                    driver.find_element_by_xpath('//a[@title="Next Page"]').click()
                    time.sleep(3)
                    tables = driver.find_elements_by_tag_name('table')
                    table = tables[1].get_attribute('outerHTML')
                    df_x = pd.read_html(table)
                    df1 = pd.concat(df_x)
                    df1 = df.append(df1)
                except:
                    break
                df1.to_excel(f'Handloom/Handloom_{str(lofi)}.xlsx')

Current output

 A | B | C
1
2
3
.
.
100
Expected Output
A | B | C
1
2
.
.
100
.
.
200
.
.
300
.
.
pandas python-3.x
1 Answers

A general approach is to create a list of dataframes, then pd.concat them once at the end. Something like this:

dfs = []

tables = driver.find_elements_by_tag_name('table')
table = tables[1].get_attribute('outerHTML')
df_i = pd.read_html(table)
df = pd.concat(df_i)
dfs.append(df)

while True:
    try:
        driver.find_element_by_xpath('//a[@title="Next Page"]').click()
        time.sleep(3)
        tables = driver.find_elements_by_tag_name('table')
        table = tables[1].get_attribute('outerHTML')
        df_x = pd.read_html(table)
        df = pd.concat(df_x)
        dfs.append(df)
    except:
        break

df = pd.concat(dfs)
df.to_excel(f'Handloom/Handloom_{str(lofi)}.xlsx')
user_131383640
Related