Scraping HTML data R studio with hidden API

user_15542064 0
Viewed 159

I am trying to generate a overview of latest crowdfunding projects with help of R studio, the website does not allow usual and straightforward Rvest (and Selector Gadget). I want to scrape the projects and details to R studio. The website has an hidden API:

https://www.geldvoorelkaar.nl/umbraco/api/ProjectApi/GetProjects

I managed to send a POST request to hidden API. Code:

url <- "https://www.geldvoorelkaar.nl/umbraco/api/ProjectApi/GetProjects"

response <- POST(url, 
     encode="json",
     add_headers(Authorization = "Token XXXXXXXXX"),
     body = "{\"texts\":[\"A simple string\"]}")

content(response)

However, I do get the error $Message [1] "The request contains an entity body but no Content-Type header. The inferred media type 'application/octet-stream' is not supported for this resource." now. I suppose that response needs more arguments? Any idea?

web-scraping google-chrome r html
1 Answers

I found the following to be working. From what I can gather, one problem was, that the required encoding was form, not json. Another problem was, that from the request you posted, you did not send any body, meaning information on what you want to get back from the API. As it is a pain to type all that out, I typically copy the curl command from chrome and paste it here to get the body formatted nicely.

If you plan on doing this frequently, I recommend putting your email adress or something in the headers of your request, so that Sysadmins have a chance of contacting you if this behavior is not wanted.

require(httr)


headers = c(
  `Email` = 'johndoe@company.com'
)

data = list(
  `Culture` = 'nl-NL',
  `Skip` = '0',
  `MaxTotal` = '18',
  `ProjectGroup` = '',
  `Filters[0][FilterFieldId]` = '6',
  `Filters[0][FilterType]` = 'Textbox',
  `Filters[1][FilterFieldId]` = '10',
  `Filters[1][FilterType]` = 'Checkbox',
  `Filters[2][FilterFieldId]` = '7',
  `Filters[2][FilterType]` = 'Radiobutton',
  `Filters[3][FilterFieldId]` = '5',
  `Filters[3][FilterType]` = 'MinMaxSliderInt',
  `Filters[3][Values][0][Id]` = '1',
  `Filters[3][Values][0][Value]` = '19200',
  `Filters[3][Values][1][Id]` = '2',
  `Filters[3][Values][1][Value]` = '2600000',
  `Filters[4][FilterFieldId]` = '1',
  `Filters[4][FilterType]` = 'MinMaxSliderDouble',
  `Filters[4][Values][0][Id]` = '1',
  `Filters[4][Values][0][Value]` = '1.2',
  `Filters[4][Values][1][Id]` = '2',
  `Filters[4][Values][1][Value]` = '9',
  `Filters[5][FilterFieldId]` = '2',
  `Filters[5][FilterType]` = 'Checkbox',
  `Filters[6][FilterFieldId]` = '3',
  `Filters[6][FilterType]` = 'MinMaxSliderInt',
  `Filters[6][Values][0][Id]` = '1',
  `Filters[6][Values][0][Value]` = '5',
  `Filters[6][Values][1][Id]` = '2',
  `Filters[6][Values][1][Value]` = '120',
  `Filters[7][FilterFieldId]` = '4',
  `Filters[7][FilterType]` = 'Checkbox',
  `Filters[8][FilterFieldId]` = '8',
  `Filters[8][FilterType]` = 'Checkbox',
  `PageId` = '1219',
  `OverviewTypeId` = '3',
  `WhiteLabelCode` = '',
  `OrderBys[0][Id]` = '5',
  `OrderBys[0][Value]` = 'Openstaand bedrag',
  `OrderByDescending` = 'false'
)

res <- httr::POST(url = 'https://www.geldvoorelkaar.nl/umbraco/api/ProjectApi/GetProjects', 
                  httr::add_headers(.headers=headers), 
                  body = data,
                  encode = "form")
result <- content(res)
user_124882170
Related