Web Scraping Table in multiple pages with same URL in r

user_19035400 0
Viewed 75

I would like to web scrape the earnings table ("RENDIMENTOS") of a stock from:

https://statusinvest.com.br/fundos-imobiliarios/rbrp11

The table has multiple pages, but the URL doesn't change.

Using the selectorGadget extension, it tells me that the name of this node is "tbody", but when reading this node, I can only see the first twelve records (first page). Is it possible to web scrape records from all pages?

I'm trying this code:

library("rvest")

url <- "https://statusinvest.com.br/fundos-imobiliarios/rbrp11"

url %>% read_html %>% 
        html_nodes("tbody") %>% 
        .[1] %>%
        html_table(fill=TRUE)

Opening the website's source code, I can see all the records on line 1377, in the following format:

<input id="results" name="results" type="hidden" value="[{"y":0,"m":0,"d":0,"ad":null,"ed":"05/08/2022","pd":"12/08/2022","et":"Rendimento","etd":"Rendimento","v":0.450000000000000000,"ov":null,"sv":"0,45000000","sov":"-","adj":false}, (.......)

Thanks

web-scraping r
1 Answers

It looks the values in the table are stored as JSON format in the value attribute of an "input" node.

So it is just a matter of locating the correct node, extracting out the attribute and converting from JSON.

library("rvest")

#read the page
url <- "https://statusinvest.com.br/fundos-imobiliarios/rbrp11"
page<- read_html(url)

#get the parent 'div' node
node <-page %>% 
   html_elements(xpath= ".//div[contains(@class, 'card chart-and-list scroll-y no-scroll-md-y rounded pt-md-3 pb-3 show-empty-callback')]") 
#get the value attribute of the input and convert from JSON
answer <- node %>% html_element("input") %>% 
                   html_attr("value")  %>% 
                   jsonlite::fromJSON()

   y m d ad         ed         pd         et        etd         v ov         sv sov   adj
1  0 0 0 NA 05/08/2022 12/08/2022 Rendimento Rendimento 0.4500000 NA 0,45000000   - FALSE
2  0 0 0 NA 07/07/2022 14/07/2022 Rendimento Rendimento 0.4500000 NA 0,45000000   - FALSE
3  0 0 0 NA 07/06/2022 14/06/2022 Rendimento Rendimento 0.4500000 NA 0,45000000   - FALSE
4  0 0 0 NA 06/05/2022 13/05/2022 Rendimento Rendimento 0.4500000 NA 0,45000000   - FALSE
5  0 0 0 NA 07/04/2022 14/04/2022 Rendimento Rendimento 0.5000000 NA 0,50000000   - FALSE
6  0 0 0 NA 08/03/2022 15/03/2022 Rendimento Rendimento 0.4200000 NA 0,42000000   - FALSE
...

I am not sure what information you are looking for, it should be here?

user_57922440
Related