Find a way to extract the URL and descriptions for images from a website

user_13589742 0
Viewed 33

So i am trying to extract the URL's and the descriptions under each picture from a website and then be able to download the pictures.

I haven't been able to get past the extraction point though.

I am using Beautifulsoup to extract the text that i want from the HTML tags and i got it to work for just the first link with the find() tag.

I started to work on getting the rest of the links with find_all() but i am getting stuck.

Can someone help me to figure out where i would need to use find_all() and do i need another for loop for the descriptions?

import requests
from bs4 import BeautifulSoup
import urllib
import csv
import time



URL = 'https://www.baps.org/vicharan'
content = requests.get(URL)

soup=BeautifulSoup(content.text, 'html.parser')


f = csv.writer(open('crawler.csv' , 'w'))
f.writerow(['description' , 'full_link'])

panelrow = soup.find('div' , {'class' : 'panelrow'})


main_class =  panelrow.find_all('div' , {'class' : 'col-xl-3 col-lg-3 col-md-3 col-sm-12 col-xs-12 padding5'})

individual_classes = panelrow.find('a' , {'class' : 'highslidooo'})

for link in individual_classes.find_all('img'):
    links=link.get('src')
    full_link = 'https://www.baps.org' + links
    description = link.get('alt')
    f.writerow([description , full_link])


print('--------------------')
print(full_link)
print(description)
beautifulsoup python-3.x
1 Answers

The class panelrow is declared on a hidden div. You should use the fullview id instead.

This will give you one <div> containing 23 <div>, each containing a <a> containing an <img>.

You were looking for the information inside the <a>, but you need to look in the <img> tags instead.

# Use the 'fullview' class 
panelrow = soup.find('div' , {'id' : 'fullview'})

main_class =  panelrow.find_all('div' , {'class' : 'col-xl-3 col-lg-3 col-md-3 col-sm-12 col-xs-12 padding5'})

# Look for 'highslide-- img-flag' links
individual_classes = panelrow.find_all('a' , {'class' : 'highslide-- img-flag'})

# Get the img tags, each <a> tag contains one
images = [i.img for i in individual_classes]

for image in images:
    src=image.get('src')
    full_link = 'https://www.baps.org' + src
    description = image.get('alt')
    f.writerow([description , full_link])

Here is the result in crawler.csv:

description,full_link
Param Pujya Mahant Swami Maharaj performs the morning arti,https://www.baps.org/Data/Sites/1/Media/GalleryImages/22118/WebImages/2020_05_20_001_Nenpur.jpg
Swamishri applies a tilak on his forehead ,https://www.baps.org/Data/Sites/1/Media/GalleryImages/22118/WebImages/2020_05_20_002_Nenpur.jpg
Swamishri applies chandlo ,https://www.baps.org/Data/Sites/1/Media/GalleryImages/22118/WebImages/2020_05_20_003_Nenpur.jpg
...
user_35487730
Related