Scrape URL recursively and store them into a list

user_15306690 0
Viewed 30

I have a for loop to scrape specific url in an HTML page, my goal is to store all URL scraped into a list.

for page in range(total_pages): #run through all pages
    url = token + str(page+1)
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'lxml') # parse page
    for link in soup.find_all('a', {'data-track': 'download'}, href=re.compile("atec")):
        all_atec = (link.get('href'))
        print(all_attec)

When I print links inside my for loop I have all the URLs:

http://www.cstb.fr/pdf/atec/GS07-H/AH191758_V2.pdf
http://www.cstb.fr/pdf/atec/GS07-H/AH191759_V2.pdf
http://www.cstb.fr/pdf/atec/GS20-U/AU14333_V1.pdf
http://www.cstb.fr/pdf/atec/GS02-C/AC2161731_V1.pdf
http://www.cstb.fr/pdf/atec/GS02-C/AC2161730_V1.pdf
http://www.cstb.fr/pdf/atec/GS02-C/AC2131565_V2.pdf

If I print links outside the loop I have only the first URL. How can I store all the URL in a list?

web-scraping beautifulsoup list python loops
1 Answers

It is simple as this:

all_urls = []
for page in range(total_pages): #run through all pages
    url = token + str(page+1)
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'lxml') # parse page
    for link in soup.find_all('a', {'data-track': 'download'}, href=re.compile("atec")):
        all_atec = (link.get('href'))
        all_urls.append(all_atec)
print(all_urls)
user_111992980
Related