R httr post with headers

user_5652614 0
Viewed 425

I have the following request in Python. How can I rewrite this in R's httr package?

The thing I'm not sure about is how to translate the json and tuple equivalent in R.

import requests

cookies = {}

headers = {
    'Host': 'mysejahtera.malaysia.gov.my',
    'Accept': 'application/json',
    'Connection': 'keep-alive',
    'Content-Length': '77',
    'Authorization': 'Basic N0ZFRkRCMTMtN0Q2MC00NEQxLUE5MTctM0',
    'Content-Type': 'application/json',
    'Accept-Language': 'en-MY;q=1, ms-MY;q=0.9',
    'User-Agent': 'MySejahtera/1.0.36 (iPhone; iOS 14.6; Scale/2.00)',
}

params = (
    ('type', 'search'),
)

data = '[{"lat":3.003090,"lng":101.678300,"classification":"LOW_RISK_NS"}]'

response = requests.post('https://mysejahtera.malaysia.gov.my/register/api/nearby/hotspots', headers=headers, params=params, cookies=cookies, data=data)
response.json()

Here's an attempt. But still gives out 400 bad request

POST("https://mysejahtera.malaysia.gov.my/register/api/nearby/hotspots?type=search",
     body = list(lat = 3.003090, lng = 101.678300, classification = 'LOW_RISK_NS'), 
     add_headers(Host= 'mysejahtera.malaysia.gov.my',
                 Accept= 'application/json',
                 Connection= 'keep-alive',
                 `Content-Length`= '77',
                 Authorization= 'Basic N0ZFRkRCMTMtN0Q2MC00NEQxLUE5MTctM0',
                 `Content-Type`= 'application/json',
                 `Accept-Language`= 'en-MY;q=1, ms-MY;q=0.9',
                 `User-Agent`= 'MySejahtera/1.0.36 (iPhone; iOS 14.6; Scale/2.00)'), verbose())
httr r python
2 Answers

The trick here seemed to be to construct the data as a list of a named list so that it is encoded to json as the API expects.

Note, most of the supplied headers proved unnecessary (or are automatically generated by httr) in order to retrieve the desired response.

library(httr)

# data as list of named list
data <- list(
  list(
    lat = 3.003090,
    lng = 101.678300,
    classification = "LOW_RISK_NS"
  )
)

params <- list(type = "search")

response <- POST("https://mysejahtera.malaysia.gov.my/register/api/nearby/hotspots",
                 query = params,
                 body = data, 
                 add_headers(Authorization = "Basic N0ZFRkRCMTMtN0Q2MC00NEQxLUE5MTctM0"),
                 encode = "json"
)
content(response)
#> $hotSpots
#> list()
#> 
#> $zoneType
#> [1] "RED"
#> 
#> $messages
#> $messages$ms_MY
#> [1] "Hai {name}, terdapat 21 kes COVID-19 dalam lingkungan radius 1km dari lokasi ini yang dilaporkan dalam masa 14 hari yang lepas."
#> 
#> $messages$en_US
#> [1] "Hi {name}, there have been 21 reported case(s) of COVID-19 within a 1km radius from your searched location in the last 14 days."
#> 
#> 
#> $note
#> NULL

Created on 2021-06-21 by the reprex package (v1.0.0)

Some Explanation

JSON from List vs List of List

See here the difference (i.e., the addition of square brackets) in the resulting strings of json between a named list and a list of a named list:

data <- list(lat = 3, lng = 101, class = "LOW")

# list
jsonlite::toJSON(data)
#> {"lat":[3],"lng":[101],"class":["LOW"]}

# list of a named list
jsonlite::toJSON(list(data))
#> [{"lat":[3],"lng":[101],"class":["LOW"]}]

Created on 2021-06-23 by the reprex package (v1.0.0)

Even Simpler Approach

Here I do more of the encoding work myself rather than letting httr take care of it as in the answer above.

Note that the square brackets surrounding the values are now missing as compared to the second (list of list) json string above. This is representative of the "unboxing" httr does.

library(httr)

response <- POST("https://mysejahtera.malaysia.gov.my/register/api/nearby/hotspots?type=search",
                 body = '[{"lat":3.003090,"lng":101.678300,"classification":"LOW_RISK_NS"}]', 
                 add_headers(Authorization = "Basic N0ZFRkRCMTMtN0Q2MC00NEQxLUE5MTctM0",
                             "Content-Type" = "application/json")
)
content(response)
#> $hotSpots
#> list()
#> 
#> $zoneType
#> [1] "RED"
#> 
#> $messages
#> $messages$ms_MY
#> [1] "Hai {name}, terdapat 22 kes COVID-19 dalam lingkungan radius 1km dari lokasi ini yang dilaporkan dalam masa 14 hari yang lepas."
#> 
#> $messages$en_US
#> [1] "Hi {name}, there have been 22 reported case(s) of COVID-19 within a 1km radius from your searched location in the last 14 days."
#> 
#> 
#> $note
#> NULL

Created on 2021-06-23 by the reprex package (v1.0.0)

user_93932130

In general, I set the input values inside a list() or a vector c() when web scraping with R.

httr's POST() accepts JSON as the body as well, as you can see on the documentation.

Your code would most likely look like this:

library(httr)

cookies <- c()

headers <- c('Host' = 'mysejahtera.malaysia.gov.my',
             'Accept' = 'application/json',
             'Connection' = 'keep-alive',
             'Content-Length' = '77',
             'Authorization' = 'Basic N0ZFRkRCMTMtN0Q2MC00NEQxLUE5MTctM0',
             'Content-Type' = 'application/json',
             'Accept-Language' = 'en-MY;q=1, ms-MY;q=0.9',
             'User-Agent' = 'MySejahtera/1.0.36 (iPhone; iOS 14.6; Scale/2.00)')

data <- list("type" = "search",
             "lat" = 3.003090,
             "lng" = 101.678300,
             "classification" = "LOW_RISK_NS")

response = POST("https://mysejahtera.malaysia.gov.my/register/api/nearby/hotspots",
                body = data,
                add_headers(.headers = headers))

I have incorporated the params into the body and I have removed the cookies, because they throw an error when they are empty.

The status code is still 400, which might indicate the need for the inclusion of other variables in the request.

user_57290950
Related