after 13th api call yfinance will not give any earnings data

Question

I have the following script to populate my database with yahoo finance information:

from multiprocessing import Pool

import json, time, yfinance

import django
django.setup()

from dividends_info.functions.stock_info import save_stock_info_data
from dividends_info.models import StockInfo

with open('tickers/nyse_tickers.json') as tickers_file:
    TICKERS = json.load(tickers_file)

TICKERS = TICKERS[0:100]

# https://www.digitalocean.com/community/tutorials/python-multiprocessing-example

def update_a_stock(ticker):
    stock, created = StockInfo.objects.get_or_create(ticker=ticker)
    yahoo_stock_obj = yfinance.Ticker(ticker.upper())
    earnings_history = yahoo_stock_obj.earnings_history
    save_stock_info_data(yahoo_stock_obj, ticker, stock, earnings_history)
    if not stock.earnings:
        print(f"No earnings for {ticker}")



def pool_handler():
    start = time.time()
    p = Pool(2)
    p.map(update_a_stock, TICKERS)

    with open("time_taken_to_populate.txt", "w") as time_file:
        time_taken = round((time.time() - start), 2)
        time_file.write(f"Time taken = {time_taken:.10f}")


if __name__ == '__main__':
    pool_handler()

The important thing here is

yahoo_stock_obj = yfinance.Ticker(ticker.upper())
earnings_history = yahoo_stock_obj.earnings_history

whereas my save_stock_info_data prints out the type of the earnings history:

ticker in save stock info func: ABEV

ticker in save stock info func: A

Saved new information for stock ABEV
Saved new information for stock A
ticker in save stock info func: ABG

ticker in save stock info func: AA

Saved new information for stock ABG
Saved new information for stock AA
ticker in save stock info func: ABM

As you can see the script starts off fine, and for the first 13 calls it will save earnings data:

python3 count_db_items.py 
21 many stocks
name: 18, summary: 18, dividends: 16, earnings: 13

After the first 13 earnings are saved, earnings_history is no longer available from the api:

ticker in save stock info func: AB

Could not find data for ACCO.
ticker in save stock info func: ACCO

Saved new information for stock AB
No earnings for AB
Saved new information for stock ACCO
No earnings for ACCO
Could not find data for ABB.
ticker in save stock info func: ABB

each earnings_history object gathered from the yahoo_stock_object of yfinance is 'None'. This happens each time the script is ran after the first 13 are saved. All the other data including dividends is available. I think dividends is sent as an array whereas earnings_history is sent as a pandas dataframe.

When I ran the script for all tickers I had 1800 stocks, 1700 dividends and only 25 results for earnings.

Running the script synchronously does not help in fact it is worse where NO earnings are saved at all:

import json, os, sys, time, yfinance

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "dividends_project.settings")

sys.path.append('../..')

import django
django.setup()

from dividends_info.functions.stock_info import save_stock_info_data
from dividends_info.models import StockInfo

# https://stackoverflow.com/questions/59159991/modulenotfounderror-no-module-named-foo-how-can-i-import-a-model-into-a-djang

# https://pythonspeed.com/articles/python-multiprocessing/

# https://github.com/pytorch/pytorch/issues/3492

f = open('tickers/nyse_tickers.json')
TICKERS = json.load(f)
f.close()

print(len(TICKERS))

TICKERS = TICKERS[:50]

def update_a_stock(ticker):
    stock, created = StockInfo.objects.get_or_create(ticker=ticker)
    yahoo_stock_obj = yfinance.Ticker(ticker.upper())
    earnings_history = yahoo_stock_obj.earnings_history
    # stock, yahoo_obj = save_stock_info_data(ticker, stock)
    save_stock_info_data(yahoo_stock_obj, ticker, stock, earnings_history)
    if not stock.earnings:
        print("earnings didn't save for this stock...the earnings are:")
        print(earnings_history)


def run():
    start = time.time()
    for ticker in TICKERS:
        update_a_stock(ticker)

    with open("time_taken_to_populate.txt", "w") as time_file:
        time_taken = round((time.time() - start), 2)
        time_file.write(f"Time taken = {time_taken:.10f}")


if __name__ == "__main__":
    run()

without the earnings data my website is pretty much useless. It doesn't seem a rate limiting issue since running a synchronous version of the script doesn't work either.

If I access an individual stock it has earnings:

import sys

ticker = sys.argv[1]

def update_a_stock(ticker):
    stock, created = StockInfo.objects.get_or_create(ticker=ticker)
    yahoo_stock_obj = yfinance.Ticker(ticker.upper())
    earnings_history = yahoo_stock_obj.earnings_history
    earnings = gather_earnings_object

in ipdb:

ipdb> earnings
[{'date': datetime.date(2022, 7, 29), 'expected': 3.31, 'actual': 3.37, 'surprise': '+1.69'}, {'date': datetime.date(2022, 4, 29), 'expected': 3.14, 'actual': 3.16, 'surprise': '+0.57'}, {'date': datetime.date(2022, 2, 2), 'expected': 3.29, 'actual': 3.31, 'surprise': '+0.73'}, {'date': datetime.date(2021, 10, 29), 'expected': 3.22, 'actual': 3.33, 'surprise': '+3.32'}, ....

I would have to manually run my script for the single api call 10000 times to populate the data with all american stocks.

How can I run an asynchronous script calling yfinance to populate my db?

after 13th api call yfinance will not give any earnings data

0 Answers