Python: Cannot fetch data from website if URL is passed as 2nd,3rd and so on list items, but works for 1st list item and direct URL how to fix this?

user_12142152 0
Viewed 78

I have CSV file containing a list of links, i stored those values in a list and then pass the list values as the URL for the page i need to get the data from and it works for the first list item but not for the list items after that. Although it works if the URL is directly passed as a string to the variable i'm sorting the URL in is there any way to fix this please help?

Here is the code:

   data = pd.read_csv("cur_hp_printer_name_links.csv")
   list_of_links = data['links'].tolist()
   link_no = 0
   specs = []
   spec = {}

   while link_no < 2:

      URL = list_of_links[link_no]
      r = requests.get(URL)

      soup = BeautifulSoup(r.content, 'html5lib')

      table = soup.find('div', attrs={'class': 'column main'})

      for rows in table.findAll('div', attrs={'class': 'page-title-wrapper product'}):
          spec['Product_name'] = rows.span.text
          specs.append(spec)
          print(spec)

      link_no += 1

here is the output

C:\Users\dhanush\Desktop\stuff\pycharm projects\venv\Scripts\python.exe" "C:/Users/dhanush/Desktop/stuff/pycharm projects/test.py"
{'Product_name': 'HP DeskJet Ink Advantage 1115 Printer'}

Process finished with exit code 0

here the part right under where the name of the product is, there's supposed to be another name which hasn't been printed

web-scraping python
1 Answers

Below is a simplified version of you code.

import requests
from bs4 import BeautifulSoup

data = []
with open('urls.txt') as f:
  urls = [line.strip() for line in f.readlines()]
  for url in urls:
    r = requests.get(url)
    if r.status_code == 200:
      soup = BeautifulSoup(r.content, 'html.parser')
      table = soup.find('div', attrs={'class': 'column main'})
      for row in table.findAll('div', attrs={'class': 'page-title-wrapper product'}):
            data.append({'product_name': row.span.text})
print(data)

urls.txt

https://store.hp.com/in-en/default/printers/hp-ink-tank-wireless-415-z4b53a.html
https://store.hp.com/in-en/default/printers/hp-deskjet-ink-advantage-1115-printer-f5s21b.html

output

[{'product_name': 'HP Ink Tank Wireless 415'}, {'product_name': 'HP DeskJet Ink Advantage 1115 Printer'}]
user_4150160
Related