how to loop over zillow/realtor using rvest to pull links

user_13977364 0
Viewed 111

I am currently using Rvest to try and pull all links from the following url: https://www.zillow.com/browse/homes/fl/miami-dade-county/ The below code satisfies what I am looking to do for one url.

#dl packages
library(tidyverse)
library(rvest)
library(xml2)
library(stringi)
library(dplyr)
library(purrr)
library(stringr)

webpage <- "https://www.zillow.com/browse/homes/fl/miami-dade-county/"
webpage <- read_html(webpage)
url_ <- webpage %>%
html_nodes("a") %>%
html_attr("href")

I am attempting to duplicate this (either a for loop or lapply) for a dataframe (called newurl) full of the same url (zillow.com/browse/homes/fl/ but each ending in a different county). I have tried both looping and lapplying but receive a different error each time. I have included my most recent error and code attempt below. Looking for advice on code to use to satisfy my need or how to edit existing. Thank you.

I have attempted many different codes but my most recent is below.

bind_rows(lapply(newurl, function(x) {
data.frame(newurl=x, toc_entry=toc <- read_html(newurl[1]) %>%
           html_nodes("a") %>%
           html_attr("href"))
})) -> toc_entries

this produces the following error: Error in UseMethod("read_xml") : no applicable method for 'read_xml' applied to an object of class "data.frame"

xml2 rvest web-scraping r
1 Answers

You can try this:

require(tidyverse)
require(rvest)
require(xml2)

start_page <- "https://www.zillow.com/browse/homes/fl/miami-dade-county/"
fl_urls <- start_page %>% 
  read_html %>% 
  html_nodes("section a") %>% 
  html_attr("href") %>% #gives you the link to all ZIP codes
  xml2::url_absolute(start_page) # convert to full URL

name_url <- function(x, base = start_page){
  data_frame(
    name = html_text(x, trim = FALSE),
    url = html_attr(x, "href") %>% 
      xml2::url_absolute(base)
  )
}

fl_urls %>% 
  head(2) %>% # Only loop the first two entries of fl_urls - remove to loop all
  map(read_html) %>% 
  map(html_nodes, "section a") %>% 
  map_df(name_url)

What results in

# A tibble: 51 x 2
   name                                 url                                            
   <chr>                                <chr>                                          
 1 1000 Hardee Rd - 1191 S Alhambra Cir https://www.zillow.com/browse/homes/fl/miami-d…
 2 1195 S Alhambra Cir - 1250 S Alhamb… https://www.zillow.com/browse/homes/fl/miami-d…
 3 1250 S Alhambra Cir APT 17 - 1410 C… https://www.zillow.com/browse/homes/fl/miami-d…
 4 1410 Mantua Ave - 1508 Zoreta Ave    https://www.zillow.com/browse/homes/fl/miami-d…
 5 1509 Delgado Ave - 1540 Zuleta Ave   https://www.zillow.com/browse/homes/fl/miami-d…
 6 1541 Cecilia Ave - 3760 Bird Rd UNI… https://www.zillow.com/browse/homes/fl/miami-d…
 7 3760 Bird Rd UNIT 513 - 4100 Salzed… https://www.zillow.com/browse/homes/fl/miami-d…
 8 4100 Salzedo St APT 604 - 420 Vitto… https://www.zillow.com/browse/homes/fl/miami-d…
 9 4210 Anderson Rd - 4420 Anderson Rd  https://www.zillow.com/browse/homes/fl/miami-d…
10 4420 Monserrate St - 4900 Suarez St  https://www.zillow.com/browse/homes/fl/miami-d…
# … with 41 more rows

Hope this helps to understand the logic of looping.

user_33013440
Related