BeautifulSoup randomly gets stuck in for loop

user_19366166 0
Viewed 61

I’ve been using BeautifulSoup to extract multiple pages of reviews from websites, and it’s worked wonders mostly but on large datasets has constantly been getting stuck at seemingly random points.

My code is always along the lines of the following.

for x in range(len(reviews)):
    reviewsoups.append(BeautifulSoup(requests.get(reviews[x]).text, ‘html.parser))

I’ve never gotten any errors or anything (except the random ConnectionReset error), but it just seems as though the loop gets stuck randomly to the point where I consistently have to interrupt the kernel (which often takes 10+ minutes to actually work) and restart the process from the index where the loop got stuck.

It seems as though in some cases, if I try and use my laptop whatsoever while the code is running (like opening Chrome etc) that aggravates the situation.

Can anyone help? It’s just incredibly irritating having to sit by my laptop waiting just in case something like this happens.

Thanks in advance.

web-scraping beautifulsoup python
1 Answers

I think I’ve found a solution.

So I was trying to ‘soup’ 9000 URLs. What I did was iteratively created variables using the globals() function, with the idea of having each variable store 100 soups, so that would be 90 variables with 100 soups each rather than one list with 9000.

I noticed that the first few hundred were very quick and then slowed down, however running 100 at a time and not constantly elongating an already massive list made a difference.

I also got no crashes.

Bear in mind I only tried this with the last 1000 or so after I got stuck at the 8000 mark, but it was so much quicker and no technical issues.

Next time I will initialise a for loop that incorporates each variable and appends eg the 1056th soup to the 11th variable as the 56th element if that makes sense.

user_193661660
Related