Multiple requests causing program to crash (using BeautifulSoup)

user_16236109 0
Viewed 370

I am writing a program in python to have a user input multiple websites then request and scrape those websites for their titles and output it. However, when the program surpasses 8 websites the program crashes every time. I am not sure if it is a memory problem, but I have been looking all over and can't find any one who has had the same problem. The code is below (I added 9 lists so all you have to do is copy and paste the code to see the issue).

import requests
from bs4 import BeautifulSoup
lst = ['https://covid19tracker.ca/provincevac.html?p=ON', 'https://www.ontario.ca/page/reopening-ontario#foot-1', 'https://blog.twitter.com/en_us/topics/company/2020/keeping-our-employees-and-partners-safe-during-coronavirus.html', 'https://www.aboutamazon.com/news/company-news/amazons-covid-19-blog-updates-on-how-were-responding-to-the-crisis#covid-latest', 'https://www.bcg.com/en-us/publications/2021/advantages-of-remote-work-flexibility', 'https://news.prudential.com/increasingly-workers-expect-pandemic-workplace-adaptations-to-stick.htm', 'https://www.mckinsey.com/featured-insights/future-of-work/whats-next-for-remote-work-an-analysis-of-2000-tasks-800-jobs-and-nine-countries', 'https://www.gsb.stanford.edu/faculty-research/publications/does-working-home-work-evidence-chinese-experiment', 'https://www.livecareer.com/resources/careers/planning/is-remote-work-here-to-stay']
for websites in range(len(lst)):
    url=lst[websites]
    cite = requests.get(url,timeout=10).content
    soup = BeautifulSoup(cite,'html.parser')
    title = soup.find('title').get_text().strip()
    print(title)
print("Didn't crash")

The second website doesn't have a title but don't worry about that

python-requests beautifulsoup crash python
1 Answers

To avoid the page from crashing, add the user-agent header to the headers= parameter in requests.get(), otherwise, the page thinks that your a bot and will block you.

cite = requests.get(url, headers=headers, timeout=10).content

In your case:

import requests
from bs4 import BeautifulSoup


headers = {
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.101 Safari/537.36"
}

lst = [
    "https://covid19tracker.ca/provincevac.html?p=ON",
    "https://www.ontario.ca/page/reopening-ontario#foot-1",
    "https://blog.twitter.com/en_us/topics/company/2020/keeping-our-employees-and-partners-safe-during-coronavirus.html",
    "https://www.aboutamazon.com/news/company-news/amazons-covid-19-blog-updates-on-how-were-responding-to-the-crisis#covid-latest",
    "https://www.bcg.com/en-us/publications/2021/advantages-of-remote-work-flexibility",
    "https://news.prudential.com/increasingly-workers-expect-pandemic-workplace-adaptations-to-stick.htm",
    "https://www.mckinsey.com/featured-insights/future-of-work/whats-next-for-remote-work-an-analysis-of-2000-tasks-800-jobs-and-nine-countries",
    "https://www.gsb.stanford.edu/faculty-research/publications/does-working-home-work-evidence-chinese-experiment",
    "https://www.livecareer.com/resources/careers/planning/is-remote-work-here-to-stay",
]
for websites in range(len(lst)):
    url = lst[websites]
    cite = requests.get(url, headers=headers, timeout=10).content
    soup = BeautifulSoup(cite, "html.parser")
    title = soup.find("title").get_text().strip()
    print(title)
print("Didn't crash")
user_123497340
Related