R web scraping: I can't pull up the elements I want

user_15260335 0
Viewed 73

I'm a beginner in web scraping using R. I'm trying to scrape the following webpage: https://bkmea.com/bkmea-members/#/company/2523.

I would like to get all text elements under div nodes with class="company_name", as well as text elements under td nodes. For example, I'm trying to fetch the company name ("MOMO APPARELS") as in the following HTML text.

<div class="comapny_header">
    <div class="company_name">MOMO APPARELS LTD</div> 
    <div class="view_all">View All</div>
</div>

So I've written the following code:

library(textreadr)
library(rvest)

companyinfo <- read_html("https://bkmea.com/bkmea-members/#/company/2523")

html_nodes(companyinfo,"div")%>% 
  html_text()  # it works

html_nodes(companyinfo,"div.company_name")%>% 
  html_text()  # doesn't work 

html_nodes(companyinfo,"td") %>% 
  html_text()  # doesn't work

If I understand correctly - the first one should pull up texts with div nodes. The second one should pull up texts within div nodes with attributes equal to company_name. The third one should pull up texts within td nodes.

The first one works (which isn't what I'm trying to get) but the second and the third ones don't - am I doing something terribly wrong?

I'd really appreciate it if you could help me out here!!

Many thanks, Sang

rvest web-scraping r html
1 Answers

The data you're looking for is retrieved by this API (it is not present in the html body) :

GET https://bkmea.com/wp-admin/admin-ajax.php?action=bkmea_get_company&id=2523

You just need to extract the id from your original url, build the url above and parse json result as following :

library(httr)

originalUrl <- "https://bkmea.com/bkmea-members/#/company/2523"
id <- sub("^.+/", "", originalUrl)
userAgent <- "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"

output <- content(GET("https://bkmea.com/wp-admin/admin-ajax.php", query = list(
        "action" = "bkmea_get_company",
        "id" = id
    ), add_headers('User-Agent' = userAgent)), as = "parsed", type = "application/json")

print(output$company$company_info$company_name)

output :

[1] "MOMO APPARELS LTD"
user_26143640
Related