R: web scraping yahoo.finance after 2019 change

user_9669381 0
Viewed 2406

I have been happily web scraping yahoo.finance pages for a long time using code largely borrowed from other stackoverflow answers and it has worked great, however in the last few weeks Yahoo has changed their tables to be collapsible/expandable tables. This has broken the code, and despite my best efforts for a few days I can't fix the bug.

Here is an example of the code that others have used for years (which is then parsed and processed in different ways by different people).

library(rvest)
library(tidyverse)

# Create a URL string
myURL <- "https://finance.yahoo.com/quote/AAPL/financials?p=AAPL"

# Create a dataframe called df to hold this income statement called df
df <- myURL %>% 
  read_html() %>% 
  html_table(header = TRUE) %>% 
  map_df(bind_cols) %>% 
  as_tibble()

Can anyone help?

EDIT FOR MORE CLARITY:

If you run the above then view df you get 

# A tibble: 0 x 0

For an example of the expected outcome, we can try another page yahoo hasn't changed such as the following:

 # Create a URL string
myURL2 <-  "https://finance.yahoo.com/quote/AAPL/key-statistics?p=AAPL"

df2 <- myURL2 %>% 
  read_html() %>% 
  html_table(header = FALSE) %>% 
  map_df(bind_cols) %>% 
  as_tibble()

If you view df2 you get a tibble of 59 observations of two variables being the main table on that page, beginning with

Market Cap (intraday)5 [value here] Enterprise value 3 [value here] And so on...

yahoo-finance rvest web-scraping r
2 Answers

This may seem a little around the houses but I wanted to avoid much of what I suspect is dynamic on the page (e.g. many of the classNames) and provide something that might have a slightly longer shelf-life.

Your code is failing, in part, because there is no table element housing that data. Instead, you can gather the "rows" of the desired output table using a more stable looking fi-row class attribute. Within each row you can then gather the columns by matching on elements with either title attribute or data-test='fin-col' based on the parent row node.

I use regex to match on the dates (as these change over time) and combine them with the static two headers to provide the final dataframe headers for output. I limit the regex to a single node's text that I know should contain pattern matches that are only those required dates.

R:

library(rvest)
library(stringr)
library(magrittr)

page <- read_html('https://finance.yahoo.com/quote/AAPL/financials?p=AAPL')
nodes <- page %>%html_nodes(".fi-row")
df = NULL

for(i in nodes){
  r <- list(i %>%html_nodes("[title],[data-test='fin-col']")%>%html_text())
  df <- rbind(df,as.data.frame(matrix(r[[1]], ncol = length(r[[1]]), byrow = TRUE), stringsAsFactors = FALSE))
}

matches <- str_match_all(page%>%html_node('#Col1-3-Financials-Proxy')%>%html_text(),'\\d{1,2}/\\d{1,2}/\\d{4}')  
headers <- c('Breakdown','TTM', matches[[1]][,1]) 
names(df) <- headers
View(df)

Sample:

enter image description here

Py:

import requests, re
import pandas as pd
from bs4 import BeautifulSoup as bs

r = requests.get('https://finance.yahoo.com/quote/AAPL/financials?p=AAPL')
soup = bs(r.content, 'lxml')
results = []

for row in soup.select('.fi-row'):
    results.append([i.text for i in row.select('[title],[data-test="fin-col"]')])

p = re.compile(r'\d{1,2}/\d{1,2}/\d{4}')
headers = ['Breakdown','TTM']
headers.extend(p.findall(soup.select_one('#Col1-3-Financials-Proxy').text))
df = pd.DataFrame(results, columns = headers)
print(df)
user_62412350

As mentioned in the comment above, here is an alternative that tries to deal with the different table sizes published. I have worked on this and have had help from a friend.

library(rvest)
library(tidyverse)

url <- https://finance.yahoo.com/quote/AAPL/financials?p=AAPL

# Download the data
raw_table <- read_html(url) %>% html_nodes("div.D\\(tbr\\)")

number_of_columns <- raw_table[1] %>% html_nodes("span") %>% length()

if(number_of_columns > 1){
  # Create empty data frame with the required dimentions
  df <- data.frame(matrix(ncol = number_of_columns, nrow = length(raw_table)),
                      stringsAsFactors = F)

  # Fill the table looping through rows
  for (i in 1:length(raw_table)) {
    # Find the row name and set it.
    df[i, 1] <- raw_table[i] %>% html_nodes("div.Ta\\(start\\)") %>% html_text()
    # Now grab the values
    row_values <- raw_table[i] %>% html_nodes("div.Ta\\(end\\)")
    for (j in 1:(number_of_columns - 1)) {
      df[i, j+1] <- row_values[j] %>% html_text()
    }
  }
view(df)
user_96693810
Related