Hi I am trying to scrape different websites but for some of them it outputs a cloudflare url . As the code below, I want the href values of the anchor tags for every tags that have a value(which is the href/a url.) and print each one of them in new lines. For example in this code:
<a href="https://www.google.com">some text</a>
its should return the https://www.google.com in output which for some websites it doesn't work at all or return the output that I mentioned below.
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
page = requests.get("https://www.swappa.com")
soup = BeautifulSoup(page.content, "html.parser")
links = soup.find_all("a", href=True)
for link in links:
print(link['href'])
and vs code output this:
https://www.cloudflare.com/?utm_source=challenge&utm_campaign=j