I am web-scraping restaurants information (e.g. names, addresses) from different websites. I generally use two main methods: the functions in the rvest() package (R), and the ones in the BeautifulSoup module (python). I manage to collect information most of the time (when R fails, generally python works) but sometimes, after reading the webpage, I can't find the information in the nodes I select. Here are two websites that give this behaviour:
Tunisia: http://www.pagesjaunes.com.tn/List?page=2
And here's the R code I use to access the restaurants' names (example is for just one page):
library(rvest)
library(xml2)
# AUSTRALIA
webpage <- read_html(x = "https://www.yellowpages.com.au/search/listings?clue=restaurant&locationClue=All+States&pageNumber=2&referredBy=www.yellowpages.com.au&&eventType=pagination")
webpage_name <- webpage %>%
html_nodes("a.listing-name") %>%
html_text(trim = TRUE)
webpage_name
# TUNISIA
webpage <- read_html(x = "http://www.pagesjaunes.com.tn/List?page=2")
webpage_name <- webpage %>%
html_nodes("div.result-info-block") %>%
html_nodes("a") %>%
html_text(trim = TRUE)
webpage_name
As you can see, the webpage_name object is empty. With python the result doesn't change:
from bs4 import BeautifulSoup
import requests
# AUSTRIALIA
webpage = requests.get("https://www.yellowpages.com.au/search/listings?clue=restaurant&locationClue=All+States&pageNumber=2&referredBy=www.yellowpages.com.au&&eventType=pagination")
soup = BeautifulSoup(webpage.content, "html.parser")
soup_names = soup.find_all("a", {"class": "listing-name"})
print(soup_names)
# TUNISIA
webpage = requests.get("www.pagesjaunes.com.tn/List?page=2")
soup = BeautifulSoup(webpage.content, "html.parser")
soup_names = soup.find_all("div", {"class": "result-info-block"})
print(soup_names)
For the case of Tunisia, I think the problem is that the URL doesn't change for the category I select, but appears just "List?page=2" for every category after the main URL. But then why I can display the list of information in the website?
For the Australian page, I really have no clue of what is going on. These are the method I generally use for every website, but sometimes I encounter these problems.
Do you have any idea on how to proceed? Thank you!