Web Scraping in R using rvest and finding the html_note

user_7128827 0
Viewed 161

I am trying to find the current html_note to fetch the replies count for each post in this forum: https://d.cosx.org/. I used CSS selector and it said .DiscussionListItem-count but it seems not working.

My code:

library(rvest)
library(tidyverse)
COS_link <- read_html("https://d.cosx.org/")
COS_link %>% 
  # The relevant tag
  html_nodes(css = '.DiscussionListItem-count') %>%      
  html_text()

I would like to fetch the replies count, for example: 1k for 1st post and 30 for 2nd post. I am wondering if I miss something or anyone has a better idea?

rvest web-scraping r
1 Answers

You can use the API and parse the json response for the title and participantCount attributes

API endpoint returning that info is:

https://d.cosx.org/api

Substring the response to remove the trailing 0 and leading ac76 then parse with a json library of choice.

Less optimal is to regex out the json string from original url

library(rvest)
library(jsonlite)
library(stringr)

url <- "https://d.cosx.org/"

r <- read_html(url) %>% 
  html_nodes('body') %>% 
      html_text() %>% 
      toString()

x <- str_match_all(r,'flarum\\.core\\.app\\.load\\((.*)\\);')  
json <- jsonlite::fromJSON(x[[1]][,2])
counts <- json$resources$attributes$participantCount

For those wishing to pair up the title with count and who don't have chinese settings a colleague helped me write the following:

library(rvest)
library(jsonlite)
library(stringr)
library(corpus)

url <- "https://d.cosx.org/"
r <- read_html(url) %>%
html_nodes('body') %>%
html_text() %>%
toString()

x <- str_match_all(r,'flarum\\.core\\.app\\.load\\((.*)\\);')
json <- jsonlite::fromJSON(x[[1]][,2])
titles <- json$resources$attributes$title
counts <- json$resources$attributes$participantCount
cf <- corpus_frame(name = titles, text = counts)
names(cf) <- c("titles", "counts")

print(cf[which(!is.na(cf$counts)),], 100)
user_62412350
Related