web scraping titles, managed to scrape links but having difficulty with file names

user_19308908 0
Viewed 27

developer_tool

How can I scrape all the names of the file for each link? I manage to scrape the links but can't seem to scrape the name of the files. For a bit more context I used beautifulsoup and find.all to scrape all the links.

web-scraping beautifulsoup python
1 Answers

Try:

import requests
from bs4 import BeautifulSoup

url = "https://m-selig.ae.illinois.edu/ads/coord_database.html"

soup = BeautifulSoup(requests.get(url).content, "html.parser")

for a in soup.select('a[href$=".dat"]'):
    link = a["href"]
    name = a.find_next_sibling(text=True).strip(" \\")
    print("{:<30} {}".format(link, name))

Prints:


...
coord/wb13535sm.dat            Woody Blanchard WB135/35 R/C sailplane airfoil
coord/wb140.dat                Woody Blanchard WB140/35/FB R/C sailplane airfoil
coord/whitcomb.dat             NASA/Langley Whitcomb integral supercritical airfoil
coord/ys900.dat                YS900 hyrdofoil (Shen / Eppler)
coord/ys915.dat                YS915 hydrofoil (Shen / Eppler)
coord/ys930.dat                YS930 hydrofoil (Shen / Eppler)
user_100359850
Related