For a project, I'm extracting hotel reviews using python. I have a list of 100+ hotels, and I'm extracting 1.500 reviews from each. The problem is that some of the hotels don't have that many reviews. What is happening is that when 1.500 is not reached the loop stops and shows an error.
Here is my code:
# The number of reviews to obtain per hotel
reviewsToGet = 1500
# Loop for all hotels
for index, row in hotelsToScrap.iterrows():
# Present feedback on which hotel is being processed
print("Processing hotel", index)
# Reset counter per hotel
reviewsExtracted = 0
# Loop until it extracts the pre-defined number of reviews
while reviewsExtracted<reviewsToGet:
# Define URL to use based on the number of reviews extracted so far
urlToUse = row['URL']
if reviewsExtracted>0:
repText = "-Reviews-or"+str(reviewsExtracted)+"-"
urlToUse = urlToUse.replace("-Reviews-",repText)
# Open and read the web page content
soup = openPageReadHTML(urlToUse)
# Process web page
hotelReviews = processPage(soup, index, hotelReviews)
# Update counter
reviewsExtracted = reviewsExtracted + 5
# Present feedback on the number of extracted reviews
print("Extracted ",reviewsExtracted,"/",reviewsToGet)
# Save the extracted reviews data frame to an Excel file
hotelReviews.to_excel("ExtractedReviewsComplete.xlsx")
What do I do to get the extraction going even if the 1.500 is not reached?