Scraping Yahoo Finance at regular intervals

user_11808231 0
Viewed 7780

I am new to web-scraping and need to scrape yahoo finance to get stock data. I was able to scrape their historical data, however the highest frequency they have is 1 day intervals, and I am looking for higher frequency (something like 30 minutes to 1 hour). Unfortunately, yahoo finance and other financial sites (from what I looked into) don't release higher frequency than 1 day for free. My solution for getting higher frequency lies in the the chart feature yahoo finance has for every stock which is shown below:

enter image description here

As you can see, as you move along the x-axis, the cross-hairs follow the data line and the legend to the top left changes the relevant information (Open, High, Low, Close, Volume, % Change). So ideally, every day I would look at the graph for the data of a company for the past 24 hours, then somehow read the legend at the left as I move a cursor from left to right.

Before starting this, I was wondering if (a) someone has already done this or (b) some useful resources to look into to make this easier.

Thank you!

yahoo-finance pandas web-scraping python
1 Answers

yfinance has a download function that uses threading for quick downloads. As mentioned in the comments, you can obtain 1m tick data but only for a 7 day period. You'll notice in the code below, if you enter a value greater than 7d, it will throw an error. 

1 Failed download:
- SPY: 1m data not available for startTime=1589678598 and endTime=1590369798. Only 7 days worth of 1m granularity data are allowed to be fetched per request.
Empty DataFrame

9:30A-4PM is market open hours for Eastern Coast. But you'll notice that some tickers trade after hours (e.g. SPY). You can set the 'prepost' attribute below to account for pre/post market hours pricing. Hope this helps. 

import yfinance as yf

data = yf.download(  # or pdr.get_data_yahoo(...
        # tickers list or string as well
        tickers = "SPY",

        # use "period" instead of start/end
        # valid periods: 1d,5d,1mo,3mo,6mo,1y,2y,5y,10y,ytd,max
        # (optional, default is '1mo')
        period = "7d",

        # fetch data by interval (including intraday if period < 60 days)
        # valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo
        # (optional, default is '1d')
        interval = "1m",

        # group by ticker (to access via data['SPY'])
        # (optional, default is 'column')
        group_by = 'ticker',

        # adjust all OHLC automatically
        # (optional, default is False)
        auto_adjust = True,

        # download pre/post regular market hours data
        # (optional, default is False)
        prepost = False,

        # use threads for mass downloading? (True/False/Integer)
        # (optional, default is True)
        threads = True,

        # proxy URL scheme use use when downloading?
        # (optional, default is None)
        proxy = None
    )

print(data)

Result:

[*********************100%***********************]  1 of 1 completed
                                 Open        High         Low       Close   Volume
Datetime
2020-05-14 09:30:00-04:00  278.950012  279.220001  278.769989  279.160004  5701148
2020-05-14 09:31:00-04:00  279.149994  279.670013  279.149994  279.640015   602618
2020-05-14 09:32:00-04:00  279.649994  279.709991  278.980011  278.980011   445239
2020-05-14 09:33:00-04:00  278.959991  279.059998  278.890015  278.970001   110324
2020-05-14 09:34:00-04:00  278.649994  278.869995  278.609985  278.780487   968209
....
....

More information: https://pypi.org/project/yfinance/

user_135086000
Related