For the most part, I've made a scraper using bs4 and a For Loop. I was wondering how to get the final contents of each "For Loop Grab" and put them into a final dataframe.
This is my current code:
from urllib.request import Request, urlopen
import bs4 as soup
from numpy.lib.utils import source
import pandas as pd
df = pd.read_excel('C:/Users/Jacob/Downloads/CEF Tickers.xlsx', sheet_name='Sheet1')
#Read in a list of tickers from a spreadsheet
tickers_list = df['Ticker'].tolist()
#final dataframe
data = pd.DataFrame(columns=tickers_list)
for ticker in tickers_list:
#Opening the website
source = urlopen(Request(f'https://www.cefconnect.com/fund/{ticker}' , headers={'User-Agent': 'Mozilla/5.0'}))
site = soup.BeautifulSoup(source,'lxml')
#Getting the exact table
table = site.find('table', attrs={'id':'ContentPlaceHolder1_cph_main_cph_main_ucPerformance_gvPast20'})
table_rows = table.find_all('tr')
#Building the df with the table
table8 = []
for tr in table_rows:
td = tr.find_all('td')
row = [tr.text for tr in td]
table8.append(row)
df8 = pd.DataFrame(table8, columns=["Period","Price Return", "NAV Return", "Category Price Return", "Category NAV Return"])
df8 = pd.DataFrame.transpose(df8)
del df8[0]
new_header = df8.iloc[0].astype(str)
df8 = df8[1:]
df8.columns = new_header
#Making the df into 1 line and then transposing it into a column
df8 = df8.unstack().to_frame().sort_index(level=1).T
df8.columns = df8.columns.map('_'.join)
df8.set_index([f'{ticker}'], inplace = True)
df8 = pd.DataFrame.transpose(df8)
data.append(df8)
Here is a sample set of tickers:
["NIE", "ZTR", "SCD", "GLV"]
When I run this I get:
KeyError: "None of ['NIE'] are in the columns"
But when I run them individually, it works.
I'm still a beginner so forgive me if I'm missing something obvious but for the life of me, I don't understand why this doesn't work.
Thank you for your time.
