I am crawling reviews from the Amazon site.
Something I felt crawling reviews.
Sometimes, you can not crawl your pages. What's wrong?
Do I have to hold a long time sleep?
Do you not crawl because of the photos in the review? I am sorry, but I exclude certain parts and crawl. I do not know this phrase. If you do not mind, can you teach me?
Is there another reason?
import re, requests, csv from bs4 import BeautifulSoup from time import sleep ##CMD : chcp 65001 def reviews_info(div): review_text = div.find("div", "a-row review-data").span.text review_author = div.find("a", "a-size-base a-link-normal author").text review_stars = div.find("div", "a-row").a.text on_review_date = div.find('span', 'a-size-base a-color-secondary review-date').text review_date = [x.strip() for x in re.sub("on ", "", on_review_date).split(",")] return { "review_text" : review_text, "review_author" : review_author, "review_stars" : review_stars, "review_date": review_date } base_url = "https://www.amazon.com/GRACE-KARIN-BoatNeck-Sleeveless-Vintage/product-reviews/B016XUCYZO/ref=cm_cr_dp_d_show_all_top?ie=UTF8&showViewpoints=1&sortBy=helpful&pageNumber=" reviews = [] NUM_PAGES = 472 for page_num in range(1, NUM_PAGES + 1): print("souping page", page_num, ",", len(reviews), " data") url = base_url + str(page_num) soup = BeautifulSoup(requests.get(url).text, 'lxml') for div in soup('div', 'a-section review'): reviews.append(reviews_info(div)) sleep(30) ################################################### # Save dict data keys = reviews[0].keys() with open('amazon_GRACE KARIN BoatNeck Sleeveless Vintage Tea Dress with Belt_review.csv', 'w', encoding="utf-8") as f: dict_writer = csv.DictWriter(f, delimiter=',', lineterminator='\n', fieldnames=keys) dict_writer.writeheader() dict_writer.writerows(reviews)