Web scraping multiple pages issue

user_13622111 0
Viewed 332

I am trying to scrape the first 4 pages from the following site for educational purposes using R code:

https://www.boliga.dk/salg/resultater?searchTab=1&page=3&sort=date-d&zipcodeFrom=1000&zipcodeTo=2499&street=

{r}
library(rvest)
library(purrr)
library(tidyverse)

url_base <- "https://www.boliga.dk/salg/resultater?searchTab=1&page=%d&sort=date-d&zipcodeFrom=1000&zipcodeTo=2499&street="

map_df(1:4,function(i){
  page <- read_html(sprintf(url_base,i))

  data.frame(Address = html_text(html_nodes(page,".font-weight-bolder.text-left")))
}) -> Address_map
View(Address_map)

The only issue with this code is that it does not actually show me the first 4 pages, but instead it provides me with the first page four times.

I hope you can help in fixing this small issue.

rvest web-scraping r
2 Answers

The page uses javascript to call an API, you can access the data, in JSON format, like this for the first page: jsonlite::fromJSON("https://api.boliga.dk/api/v2/sold/search/results?searchTab=1&page=1&sort=date-d&zipcodeFrom=1000&zipcodeTo=2499&street=")

To put together all 4 pages, try this: 

library(dplyr)
library(jsonlite)

url_base <- "https://api.boliga.dk/api/v2/sold/search/results?searchTab=1&page=%d&sort=date-d&zipcodeFrom=1000&zipcodeTo=2499&street="

Address_map <-lapply(1:4,function(i){
  print(i)
  # pause to prevent attacking the server
  Sys.sleep(2)
  #retrieve the results data frame from the returned data structure
  fromJSON(sprintf(url_base, i))$result
}) 

#bind everything together
answer <- bind_rows(Address_map)
user_57922440

Look what happens if you paste the result of sprintf(url_base,2) into a browser, it loads the website at table 1. So I think you cannot scrape this data with rvest, you need a package like RSelenium to scrape this data. With RSelenium you can access the website and then move to the second table etc.

user_62421320
Related