Scrape several links stored inside a unordered list from several webpage using a for loop

Viewed 55

I am currently scraping the the website from the czech republic parliament. In general, I am interested in the names of deputies which are members of different committees. For this I want to extract links from several overview pages. An example is the following website: https://www.psp.cz/sqw/hp.sqw?k=508&o=7 There are 16 websites like this which I have downloaded by the following code:

for (i in 1:length(subcommittee_links_2013_2017_list)) {
  if (!file.exists(paste0(folder, subcommittee_names_2013_2017[i]))) {
    download.file(subcommittee_links_2013_2017_list[i], destfile = paste0(folder, subcommittee_names_2013_2017[i]))
    Sys.sleep(runif(1, 5, 7))
  }
}

Those documents have all the same node structure:

<html class="no-js translated-ltr" lang="en">
<body>
  <!-- Header -->
  <div id="header">
  <!-- Header -->    
  <hr>
  <hr>
  <!-- Body -->   
  <div id="body"> 
     <div class="part-content">
       <!-- Main content -->
       <div id="main-content">
         <h1>…<h1>
         <b>…<b>
         <font style="vertical-align: inherit;"<…>/font>
         <ul>
           <li>
             ::marker
             <a href="hp.sqw?k=522&o=7">
               <font style="vertical-align: inherit;">
             </a>
           <li>
           <li>
             ::marker
             <a href="hp.sqw?k=523&amp;o=7">
               <font style="vertical-align: inherit;">
             </a>
           <li>
         </ul>
         </br>  
       </div>

This example documents have two links stored in the unordered list. I have documents which have more than two links but the general node structure is the same. Those links I want to extract and store them in a list such that one list entry corresponds to one document (so 16 list entries in the end). To extract the links I tried the following for loop.

subcommittee_links <- list()
for (i in 1:length(list_files_path)) {
  html_out <- read_html(list_files_path[i], encoding="windows-1250")
  subcommittee_links[i] <- html_nodes(html_out, xpath = '/html/body/div[2]//ul//a') %>% html_attrs()
}

The for loop results in a list which indeed contains 16 entries (one per document) but each entry consist only of the first link.

subcommittee_links
[[1]]
               href 
"hp.sqw?k=3520&o=7" 

[[2]]
               href 
"hp.sqw?k=3620&o=7" 

[[3]]
               href 
"hp.sqw?k=3821&o=7" 
.
.
. 

For example, the first list entry should consist of four links instead of one.

I tried many different xpath and css selector variations but I think the problem is the loop rather than the xpath.

Thank you very much in advance!

1 Answers

I think it is because i is always length 1. So you should really be seeing number of items to replace is not a multiple of replacement length.

You could wrap the return in list() e.g.

subcommittee_links[i] <- list(html_nodes(html_out, xpath = '/html/body/div[2]//ul//a') %>% html_attrs())
Related