Question about Yfinance, IndexError, and Numpy arrays

Question

I am trying to use linear regression using data pulled from yfinance to predict future stock prices, but I am having trouble using linear regression after transposing my data's shape.

Here I create a normalization function

def normalize_data(df):
    # df on input should contain only one column with the price data (plus dataframe index)
    min = df.min()
    max = df.max()
    x = df 
    
    # time series normalization part
    # y will be a column in a dataframe
    y = (x - min) / (max - min)
    
    return y

And another function to pull stock prices from Yfinance that calls the normalization function

def closing_price(ticker):
    #Asset = pd.DataFrame(yf.download(ticker, start=Start,end=End)['Adj Close'])   
    Asset = pd.DataFrame(yf.download(ticker, start='2022-07-13',end='2022-09-16')['Adj Close'])   
    Asset = normalize_data(Asset)
    return Asset.to_numpy()

I then pull 11 different stocks using the function

MRO= closing_price('MRO')
HES= closing_price('HES')
FANG= closing_price('FANG')
DVN= closing_price('DVN')
PXD= closing_price('PXD')
COP= closing_price('COP')
CVX= closing_price('CVX')
APA= closing_price('APA')
EOG= closing_price('EOG')
HAL= closing_price('HAL')
BLK = closing_price('BLK')

Which works so far But when I try to merge the first 10 numpy arrays together,

X = np.array([MRO, HES, FANG, DVN, PXD, COP, CVS, APA, EOG, HAL])[:, :, 0]
X = np.transpose(X)

it gives me the error for the first line when I merge the numpy arrays

:1: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray

Draenth · Answer

Have you tried passing the following as is suggested by your error message?

X = np.array([MRO, HES, FANG, DVN, PXD, COP, CVS, APA, EOG, HAL], dtype=float)[:, :, 0]

Alternatively, what are you trying to do with your data afterwards, run a linear regression? Does the data have to be an np array? Often working with data is a lot easier using pandas.DataFrame, and basically all machine learning libraries such as sklearn or statsmodels or any other you might want to use will have pandas support.

To create one big dataset out of these you could try the following:

data = pd.DataFrame() #creating empty dataframe
list_of_tickers = [MRO, HES, FANG, DVN, PXD, COP, CVS, APA, EOG, HAL, BLK]

for ticker in list_of_tickers:
    for column in ticker: #because each column will just be labelled "Adj. Close" and you can't name multiple columns the same way
        new_name = str(ticker) + "_" + str(column) #columns in data will then be named "MRO_Adj. Close", "HES_Adj. Close", etc
        ticker[new_name] = ticker[column]
        ticker = ticker.drop(column, axis=1) 
 
    data = pd.concat([data, ticker], axis=1)

Additionally, this neatly prevents problems that might arise from issues that different stock tickers have or lack different dates in their dataset, as was correctly pointed out by Kevin Choon Liang Yew in the comments above.

Question about Yfinance, IndexError, and Numpy arrays

1 Answers