Applying a function to multiple lists

user_19445070 0
Viewed 31

I am doing research on U.S. Lobbying, who publishes their data as an open API that is very poorly integrated and only seems to allow 250 observations to be downloaded at one time. I would like to compile the whole data set into one data table but am struggling with the last step to do so. This is what I have thus far

base_url <- sample("https://lda.senate.gov/api/v1/contributions/?page=", 10, rep = TRUE) #Set the number between the commas as how many pages you want
numbers <- 1:10 #Set the second number as how many pages you want
pagesize <- sample("&page_size=250", 10, rep = TRUE) #Set the number between the commas as how many pages you want

pages <- data.frame(base_url, numbers, pagesize)
pages$numbers <- as.character(pages$numbers)
pages$url <- with(pages, paste0(base_url, numbers, pagesize)) # creates list of pages you want. the list is titled pages$url

for (i in 1:length(pages$url)) assign(pages$url[i], GET(pages$url[i])) # Creates all the base lists in need of extraction

The last two things I need to do are extract the data table from the created lists and then full join all of them. I know how to join all of them but extracting the data frames is proving to be challenging. basically, to all the created lists I need to apply the function fromJSON(rawToChar(list$content)). I have tried using lapply but have yet to figure it out. any help would be greatly welcomed!

lapply r
1 Answers

When you were assigning GET(pages$url[i])) to your data frame you were coercing it to a character vector. Better to assign it to a list and keep it as a response:

library(httr)
library(jsonlite)
library(dplyr) # for bind_rows
page_content <- list()
for (i in 1:length(pages$url)) page_content[[i]] <- GET(pages$url[i]) # Creates all the base lists in need of extraction

Then you can use the code you had written - fromJSON(rawToChar()) - to extract it from raw bytes to characters:

results_list <- lapply(
    page_content,
    \(page) fromJSON(rawToChar(page[["content"]]))["results"][[1]]
)

results_table <- do.call(bind_rows, results_list)

dim(results_table) # 2500 27

names(results_table)
#  [1] "url"                          "filing_uuid"                  "filing_type"                  "filing_type_display"          "filing_year"
#  [6] "filing_period"                "filing_period_display"        "filing_document_url"          "filing_document_content_type" "filer_type"
# [11] "filer_type_display"           "dt_posted"                    "contact_name"                 "comments"                     "address_1"
# [16] "address_2"                    "city"                         "state"                        "state_display"                "zip"
# [21] "country"                      "country_display"              "registrant"                   "lobbyist"                     "no_contributions"
# [26] "pacs"                         "contribution_items"
user_125450410
Related