Web page doesn't render using python requests and BeautifulSoup

user_10574250 0
Viewed 187

I am trying to scrape the text from this webpage using the requests and BeautifulSoup module in python: https://seekingalpha.com/article/4441901-apple-inc-aapl-ceo-tim-cook-on-q3-2021-results-earnings-call-transcript

However once the request has been made and I try and get the text using the following code, I only get the following text:

url = "https://seekingalpha.com/article/4441901-apple-inc-aapl-ceo-tim-cook-on-q3-2021-results-earnings-call-transcript"
head={'YOUR HEADER PARAMETERS'}
response = requests.get(url, headers=head)
transcript = BeautifulSoup(response.content, 'html.parser').text

The text:

Apple Inc. (AAPL) CEO Tim Cook on Q3 2021 Results - Earnings Call Transcript | Seeking Alpha


Javascript is Disabled
Your current browser configurationis not compatible with this site.

Is there any way I can get round this to get the text from the webpage?

Thanks

python-requests web-scraping beautifulsoup python
1 Answers

The data in the URL is being loaded by an API. I have tried to get the title and content.

Here is the API endpoint. You can make a GET request to this end-point and find the data you need.

https://seekingalpha.com/api/v3/articles/4441901?include=author%2CprimaryTickers%2CsecondaryTickers%2CotherTags%2Cpresentations%2Cpresentations.slides%2Cauthor.authorResearch%2Cco_authors%2CpromotedService%2Csentiments

The content (I mean the variable content) is in HTML format, you can use beautifulsoup to extract whatever data you need.

Here is the code that prints the title and some content.

import requests
url = 'https://seekingalpha.com/api/v3/articles/4441901?include=author%2CprimaryTickers%2CsecondaryTickers%2CotherTags%2Cpresentations%2Cpresentations.slides%2Cauthor.authorResearch%2Cco_authors%2CpromotedService%2Csentiments'

r = requests.get(url)
r = r.json()
title = r['data']['attributes']['title']
content = r['data']['attributes']['content']

print(title)
print(content[:500].strip())

Apple Inc. (AAPL) CEO Tim Cook on Q3 2021 Results - Earnings Call Transcript

<p>Apple Inc. <span class="ticker-hover-wrapper">(NASDAQ:<a href="https://seekingalpha.com/symbol/AAPL" title="Apple Inc.">AAPL</a>)</span> Q3 2021 Earnings Conference Call July 27, 2021  5:00 PM ET</p>
<p><strong>Company Participants</strong></p>
<p>Tejas Gala - Director of IR and Corporate Finance</p>
<p>Tim Cook - CEO</p>
<p>Luca Maestri - CFO</p>
<p><strong>Conference Call Participants</strong></p>
<p>Chris Caso - Raymond James</p>
<p>Jim Suva - Citigroup</p>
<p>Shannon Cross - Cross Resea
.
.
.
user_27732060
Related