rvest: language selection not working in tripadvisor

user_8053817 0
Viewed 230

I am facing a web scraping problem. I intend to scrape few comments on tripadvisor. I would like to use rvest and to get comments in all languages. From this questions I understood that a possible way was to use ?filterLang=ALL at the end of the url. In a web browser, it does work. Example:

https://www.tripadvisor.com/Restaurant_Review-g187147-d2013853-Reviews-114_Faubourg-Paris_Ile_de_France.html?filterLang=ALL

Does provide comments with "All languages" selected (and you can see a lot of french comments). Here is my problem: I try to get comment' titles:

library(rvest)
url <- "https://www.tripadvisor.com/Restaurant_Review-g187147-d2013853-Reviews-114_Faubourg-Paris_Ile_de_France.html?filterLang=ALL"

reviews_html <- read_html(url)

reviews_html %>%
  html_nodes(xpath = "//span[@class='noQuotes']") %>%
  html_text()

 [1] "I've never visited this restaurant," "Perfect"                            
 [3] "Memorable experience"                "Tasty"                              
 [5] "Absolutely spectacular"              "Excellent"                          
 [7] "Wonderfullll"                        "A Perfect Evening"                  
 [9] "Dinner "                             "Perfect dinner and evening"

I only got the English ones. And here the weird thing: if I try to get the number of pages:

reviews_html %>%
  html_nodes(xpath = "//div[@data-tab='TABS_REVIEWS']//a[@data-page-number]")%>%
  html_text()

[1] "Next" "1"    "2"    "3"    "4"    "5"    "6"    "176"

I have the number of comment pages corresponding to "All languages" selection! If you compare with the case without language selection

url <- "https://www.tripadvisor.com/Restaurant_Review-g187147-d2013853-Reviews-114_Faubourg-Paris_Ile_de_France.html"

reviews_html <- read_html(url)

reviews_html %>%
  html_nodes(xpath = "//span[@class='noQuotes']") %>%
  html_text()

 [1] "I've never visited this restaurant," "Perfect"                            
 [3] "Memorable experience"                "Tasty"                              
 [5] "Absolutely spectacular"              "Excellent"                          
 [7] "Wonderfullll"                        "A Perfect Evening"                  
 [9] "Dinner "                             "Perfect dinner and evening"

I get the same comments, but:

reviews_html %>%
  html_nodes(xpath = "//div[@data-tab='TABS_REVIEWS']//a[@data-page-number]")%>%
  html_text()

[1] "Next" "1"    "2"    "3"    "4"    "5"    "6"    "61"

I get the number of pages corresponding to the English language selection. I tried setting also the cookies:

library(httr)

url <- "https://www.tripadvisor.com/Restaurant_Review-g187147-d2013853-Reviews-114_Faubourg-Paris_Ile_de_France.html?filterLang=ALL"
httr::GET(url, 
          set_cookies(`TALanguage` = "ALL",
                      `Domain` = ".tripadvisor.com"))%>%
  read_html()%>%
  html_nodes(xpath = "//span[@class='noQuotes']") %>%
  html_text()

But it did not work either. Does anyone understand what is going on, and what I could do to actually get the comments in all languages with rvest ?

rvest web-scraping r
1 Answers

When you select the filter manually, there is a POST call on the same url. Setting filterLang=ALL in the form body correctly returns the data :

library(rvest)
library(httr)

reviews_html <- POST(
    "https://www.tripadvisor.com/Restaurant_Review-g187147-d2013853-Reviews-114_Faubourg-Paris_Ile_de_France.html",
    add_headers('x-requested-with'= 'XMLHttpRequest'),
    body = list(
      preferFriendReviews = "FALSE",
      t = "",
      q = "", # filter by mention, try "france"
      filterSeasons = "", # "1" is mar-may / "2" is jun-aug / "3" is sep-nov / "4" is dec-feb
      filterLang = "ALL", # try "zhCN" or "fr"
      filterSafety = "FALSE",
      filterSegment = "", # "3" is families / "2" is couples / "5" is solo / "1" is business / "4" is friends
      trating = "", # stars: "5" / "4" / "3" / "2" / "1" / "0"
      isLastPoll = "false",
      changeSet = "REVIEW_LIST"
    ), 
    encode = "form") %>%
    read_html()

reviews <- reviews_html %>%
    html_nodes(xpath = "//span[@class='noQuotes']") %>%
    html_text()

print(reviews)

pages  <- reviews_html %>%
  html_nodes(xpath = "//div[@data-tab='TABS_REVIEWS']//a[@data-page-number]")%>%
  html_text()

print(pages)

In the above code, I've added some description about the fields if you need those filters

kaggle link

Output:

 [1] "I've never visited this restaurant," "Excellente expérience"              
 [3] "Du grand art"                        "Promesse tenue"                     
 [5] "Une soirée de rêve en famille"       "Délicieux !!! "                     
 [7] "Une expérience inoubliable"          "UN CERTAIN REGARD"                  
 [9] "Excellent soiree en couple"          "Une soirée magnifique"              
[1] "Next" "1"    "2"    "3"    "4"    "5"    "6"    "176"
user_26143640
Related