Beautifulsoup selects tag to Pandas dataframe

Question

I am currently working on my quant project. Here is the thing I am expriencing: I need first-hand stock data to work with, I have a way of downloading stock csv files, however not the best way, because I don't wanna bomb the servers, let's say every two or three days with 1GB downloading.

What I have is a collection of all data of 5 year span. I wanna write a code that can update new records and pd.concat() it with the old ones, which is more effient way to do it.

this is the url and code i have:

url = 'http://quotes.money.163.com/service/chddata.html?code=1000001&start=20200101&end=20200401'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) '
                  'AppleWebKit/537.36 (KHTML, like Gecko) '
                  'Chrome/63.0.3239.132 Safari/537.36 QIHU 360SE'
}
response = requests.get(url, headers=headers).content
soup = bs4.BeautifulSoup(response, 'lxml')
content = soup.select('p')[0]

and it returns:

日期,股票代码,名称,...,流通市值,成交笔数
2020-04-01,000001,平安银行,...,2.50142285572e+11,2.50140272863e+11,32484
2020-03-31,000001,平安银行,...,2.48395752934e+11,2.48393754278e+11,32700
2020-03-30,000001,平安银行,...,2.51112581482e+11,2.51110560966e+11,37324
<..>
2020-01-06,000001,平安银行,...,3.3125902364e+11,3.31256358245e+11,48856
2020-01-03,000001,平安银行,...,3.33393674642e+11,3.33390992071e+11,58875
2020-01-02,000001,平安银行,...,3.2737784e+11,3.27375205834e+11,67692

my next line of code strucks me with troubles:

df1 = pd.DataFrame(content.prettify())[0]

or with another approach:

soup = bs4.BeautifulSoup(response, 'lxml').get_text()
df1 = pd.DataFrame(soup)

returns:

ValueError: DataFrame constructor not properly called!

I am not quite sure how to fix this issue. Would you please take a look at it? Thank you.

my alternative way would be downloading all recently records to my HDD, can concat them.

user_6858244 · Answer

You can read data directly from online CSV


import pandas as pd

URI = 'http://quotes.money.163.com/service/chddata.html?code=1000001&start=20200101&end=20200401'

df = pd.read_csv(URI, encoding='euc_kr')

I am not sure which encoding is used without testing but this code should return the values.

Beautifulsoup selects tag to Pandas dataframe

1 Answers