using BeautifulSoup to grab data from a table

user_15606528 0
Viewed 87

I've read through countless other posts and tried a lot of techniques, but I can't seem to get the data I want from a table on the below website. I can only return other divs and their classes, but not the values.

I am looking to get all the rows from the three columns (by airline, by origin airport, by destination airport) here: https://flightaware.com/live/cancelled

I've tried searching for the 'th class' but it only returns the div information and not the data.

Any help is appreciated

Thank you

my attempt:

rows = soup.findAll('table', attrs={'class': 'cancellation_boards'})
for r in rows:
    t = r.find_all_next('div', attrs= {'class':'cancellation_board'})
for r in rows:
    r.text
beautifulsoup python
2 Answers

The data you see is loaded via Ajax request so BeautifulSoup doesn't see it. You can simulate it via requests. To load the data to one big dataframe, you can use next example:

import requests
import pandas as pd


url = "https://flightaware.com/ajax/airport/cancelled_count.rvt"

params = {
    "type": "airline",
    "timeFilter": "((b.sch_block_out BETWEEN '2022-04-02 8:00' AND '2022-04-03 8:00') OR (b.sch_block_out IS NULL AND b.filed_departuretime BETWEEN '2022-04-02 8:00' AND '2022-04-03 8:00'))",
    "timePeriod": "today",
    "airportFilter": "",
}

all_dfs = []
for params["type"] in ("airline", "destination", "origin"):
    df = pd.read_html(requests.get(url, params=params).text)[0]
    df["type"] = params["type"]
    all_dfs.append(df)

df_final = pd.concat(all_dfs)
print(df_final)
df_final.to_csv("data.csv", index=False)

Prints:

                              Airline                                                   Airport Cancelled      Delayed              type
                              Airline                                                   Airport         #    %       #    %             
0                       China Eastern                                                       NaN       509  45%      28   2%      airline
1                     Spring Airlines                                                       NaN       443  82%       5   0%      airline
2                           Southwest                                                       NaN       428  12%    1369  39%      airline
3                   American Airlines                                                       NaN       317  10%     472  16%      airline
4                               Delta                                                       NaN       229   8%     444  16%      airline
5                              Spirit                                                       NaN       190  23%     207  26%      airline
6                     Hainan Airlines                                                       NaN       167  41%       9   2%      airline
7                             JetBlue                                                       NaN       144  14%     494  48%      airline
8                            Lion Air                                                       NaN       129  20%      53   8%      airline
9                             easyJet                                                       NaN       121   8%     471  32%      airline

...

and saves data.csv (screenshot from LibreOffice):

enter image description here

user_100359850

As the url is dynamic, you also can grab table data pandas with selenium.

import time
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import pandas as pd
driver = webdriver.Chrome(ChromeDriverManager().install())

url ="https://flightaware.com/live/cancelled"
    
driver.maximize_window()
driver.get(url)
time.sleep(5)


soup = BeautifulSoup(driver.page_source, 'lxml')
table=soup.select_one('table.cancellation_boards')
#driver.close()

df = pd.read_html(str(table),header=0)[0]
print(df)

Output:

 By airline Unnamed: 1 By origin airport Unnamed: 3         By destination airport
0    Cancelled  Cancelled           Delayed    Delayed                        Airline
1            #          %                 #          %                        Airline
2    Cancelled  Cancelled           Delayed    Delayed                        Airport
3            #          %                 #          %                        Airport
4    Cancelled  Cancelled           Delayed    Delayed                        Airport
..         ...        ...               ...        ...                            ...
308         10         3%                55        17%    Luis Munoz Marin Intl (SJU)
309         10         3%               126        39%    Geneva Cointrin Int'l (GVA)
310          9         2%                33        10%                   Sydney (SYD)
311          9        11%                17        21%              Punta Gorda (PGD)
312          9         3%                10         3%  Chengdu Shuangliu Int'l (CTU)     

[313 rows x 5 columns]
user_128484110
Related