I am currently scraping the the website from the czech republic parliament. In general, I am interested in the names of deputies which are members of different committees. For this I want to extract links from several overview pages. An example is the following website: https://www.psp.cz/sqw/hp.sqw?k=508&o=7 There are 16 websites like this which I have downloaded by the following code:
for (i in 1:length(subcommittee_links_2013_2017_list)) {
if (!file.exists(paste0(folder, subcommittee_names_2013_2017[i]))) {
download.file(subcommittee_links_2013_2017_list[i], destfile = paste0(folder, subcommittee_names_2013_2017[i]))
Sys.sleep(runif(1, 5, 7))
}
}
Those documents have all the same node structure:
<html class="no-js translated-ltr" lang="en">
<body>
<!-- Header -->
<div id="header">
<!-- Header -->
<hr>
<hr>
<!-- Body -->
<div id="body">
<div class="part-content">
<!-- Main content -->
<div id="main-content">
<h1>…<h1>
<b>…<b>
<font style="vertical-align: inherit;"<…>/font>
<ul>
<li>
::marker
<a href="hp.sqw?k=522&o=7">
<font style="vertical-align: inherit;">
</a>
<li>
<li>
::marker
<a href="hp.sqw?k=523&o=7">
<font style="vertical-align: inherit;">
</a>
<li>
</ul>
</br>
</div>
This example documents have two links stored in the unordered list. I have documents which have more than two links but the general node structure is the same. Those links I want to extract and store them in a list such that one list entry corresponds to one document (so 16 list entries in the end). To extract the links I tried the following for loop.
subcommittee_links <- list()
for (i in 1:length(list_files_path)) {
html_out <- read_html(list_files_path[i], encoding="windows-1250")
subcommittee_links[i] <- html_nodes(html_out, xpath = '/html/body/div[2]//ul//a') %>% html_attrs()
}
The for loop results in a list which indeed contains 16 entries (one per document) but each entry consist only of the first link.
subcommittee_links
[[1]]
href
"hp.sqw?k=3520&o=7"
[[2]]
href
"hp.sqw?k=3620&o=7"
[[3]]
href
"hp.sqw?k=3821&o=7"
.
.
.
For example, the first list entry should consist of four links instead of one.
I tried many different xpath and css selector variations but I think the problem is the loop rather than the xpath.
Thank you very much in advance!