The WebScraper is not running

Hellrockerz 0
Viewed 27

I am using BeautifulSoup for scraping the Nasdaq site. Using the code below,

from urllib import request
from bs4 import BeautifulSoup
import requests
import urllib.request, urllib.error, urllib.parse
import xml.etree.ElementTree as ET
import ssl

ctx = ssl.create_default_context()
ctx.check_hostname= False
ctx.verify_mode = ssl.CERT_NONE

sym = input('Enter Ticker: ').upper()
ur = 'https://www.nasdaq.com/feed/rssoutbound?symbol='
url = ur+sym

html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html.parser')
print(soup)

The problem is the the moment I run this code, it hangs. Like it doesn't do anything just stops there. I was entering the ticker as msft for microsoft. I need to know if my code has any error. And if not, then why the code is not running.

urllib xml web-scraping python
1 Answers

Just add headers:

headers = {
  'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}

Full code:

import requests

sym = input('Enter Ticker: ')
url = f"https://www.nasdaq.com/feed/rssoutbound?symbol={sym}"

headers = {
  'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}

response = requests.get(url, headers=headers)

print(response.text)
userXeFr0
Related