Extract data for all the instances from a dynamic website

user_6327771 0
Viewed 58

I'm trying to scrape the data on all gym locations from https://www.xercise4less.co.uk/find-a-gym/.

In Developer Tools I found a pointer to the Web API URL that should store this information under https://www.xercise4less.co.uk/Umbraco/Api/FindAGymApi/GetAll but when I run it in the browser I get

The 'ObjectContent`1' type failed to serialize the response body for content type 'text/xml; charset=utf-8'

Similarly, if I run the following code:

# user_agent argument is optional here and results are the same whether I include it or not 
httr::GET('https://www.xercise4less.co.uk/Umbraco/Api/FindAGymApi/GetAll',  httr::user_agent("httr"))

Any ideas on how to go about this?

Alternatively, I can (almost) access all the gym IDs by 

library(rvest)
library(magrittr)

url <- "https://www.xercise4less.co.uk/find-a-gym/"
my_pg <- read_html(url) 
my_pg %>% html_nodes('select > option')

But then I'm still not sure about how to iterate over all the IDs in order to get the complete list of coordinates/locations. Thanks for any pointers.

rvest httr web-scraping r
2 Answers

You are pretty much there you just need to set the right request header expected by server then you get all the info for all the gyms.

library(httr)

headers = c('Accept'='application/json, text/javascript, */*; q=0.01')
r <- content(httr::GET(url = 'https://www.xercise4less.co.uk/Umbraco/Api/FindAGymApi/GetAll', httr::add_headers(.headers=headers)))
print(r)
user_62412350

This code should get you what you need, or at least closer to it. I piggybacked off of your second example. I would probably remove the first row. 

final <- as.character(my_pg %>% html_nodes('select > option'))
df <- data.frame(do.call(rbind, strsplit(final, '>', fixed=TRUE)), stringsAsFactors = FALSE)
df$X1 <-sapply(strsplit(df$X1, '=', fixed=TRUE), "[", 2 )
df$X1 <- gsub('[\"]', '', df$X1)

df$X2 <-sapply(strsplit(df$X2, '<', fixed=TRUE), "[", 1 )


df = subset(df, select = -c(X3) )

output

    X1                             X2
1 <NA>              Select a location
2 1104        Xercise4Less Bolton Gym
3 1248      Xercise4Less Bradford Gym
4 1249 Xercise4Less Brierley Hill Gym
5 1250       Xercise4Less Bristol Gym
6 1251       Xercise4Less Burnley Gym
user_116059970
Related