Scraping data table that doesn't exist in page source

user_3871071 0
Viewed 145

I want to scrape the data table on this website.

I check the page source for this page, the table doesn't exist in the page source.

Then I checked the network information when refresh the website, it seems the data table is obtained by sending POST request to this url:　

http://datacenter.mep.gov.cn:8099/ths-report/report!list.action

Then I tried to send POST request, but just got nothing with status 500.

I was wondering is there anyway of scraping this table down by using R?

Thanks.

web-scraping web-crawler r post
1 Answers

Good sleuthing!

It was making GET request for me. This seems to do the trick. It also tries to pick the proper target for you:

library(httr)
library(rvest)
library(stringi)

pg <- read_html("http://datacenter.mep.gov.cn/index!MenuAction.action?name=259206fe260c4cf7882462520e1e3ada")

html_nodes(pg, "div[onclick]") %>% 
  html_attr("onclick") %>% 
  stri_replace_first_fixed('load("', "") %>% 
  stri_replace_last_regex('",".*$', "") -> report_urls

head(report_urls)
## [1] "http://datacenter.mep.gov.cn:8099/ths-report/report!list.action?xmlname=1462849093743"
## [2] "http://datacenter.mep.gov.cn:8099/ths-report/report!list.action?xmlname=1462764947052"
## [3] "http://datacenter.mep.gov.cn:8099/ths-report/report!list.action?xmlname=1465594312346"
## [4] "http://datacenter.mep.gov.cn:8099/ths-report/report!list.action?xmlname=1462844293531"
## [5] "http://datacenter.mep.gov.cn:8099/ths-report/report!list.action?xmlname=1462844935563"
## [6] "http://datacenter.mep.gov.cn:8099/ths-report/report!list.action?xmlname=1462845592195"

rpt_pg <- read_html(report_urls[1])
html_table(rpt_pg)[[2]]
# SO won't let me paste the table
user_14570510
Related