Python -- Beautiful Soup -- Returning Information if a tag is empty or has values

Question

I decided to learn Python since I now have more time (due to pandemic) and have been teaching myself Python.

I am trying to scrape tax rates from a site and can get almost everything I need. Below is a snippet of the code that comes out of my Soup variable as well as the relevant piece of Python.

Where I am having difficulty is I am finding the option tag along with the data-alias that is empty (""). However, if you look at the code below there are some data-alias stages that are not empty (see UAE or Great Britain)- they have some countries listed.

I am looking to get the data-url and country name from these as well.

How do I code this to get empty tags and non-empty tags as I am losing some required information?

Thanks, Seth

My code:

import requests
from bs4 import BeautifulSoup
import re

l=[]
r = requests.get("https://taxsummaries.pwc.com/")
c=r.content
soup = BeautifulSoup(c, "html.parser")
all = soup.find_all("option", {"data-alias":""})

Website Information:

user_12149587 · Answer

This is one way, which would also introduce you to some other Python topics of potential interest (as you mentioned you are learning):

all = soup.find_all("option", {"data-alias":True})

results=[]
for item in all:
   results.append({item.text:item['data-url']})

Comments:

soup.find_all("option", {"data-alias":True}) - this will find all option tags having a data-alias attribute, irrespective of its value
results is a list to add the records in scope to (results.append)
{item.text:item['data-url']} - this is a dictionary where the key is the country name (item.text) and the value is data-url attribute (item['data-url'])

An equivalent (and more compact) way of writing the results list is using the concept of list comprehensions.

results=[{item.text:item['data-url']} for item in all]

user_9501508 · Answer

You need to use {"data-alias":True}. You can try it:

import requests
from bs4 import BeautifulSoup
l=[]
r = requests.get("https://taxsummaries.pwc.com/")
c=r.content
soup = BeautifulSoup(c, "html.parser")
options = soup.find_all('option', {"data-alias":True})
for each in options:
    print("country_name : " + str(each.text), " data-url : " + str(each['data-url']))

Output will be:

country_name : Albania  data-url : https://taxsummaries.pwc.com:443/albania
country_name : Algeria  data-url : https://taxsummaries.pwc.com:443/algeria
country_name : Angola  data-url : https://taxsummaries.pwc.com:443/angola
country_name : Argentina  data-url : https://taxsummaries.pwc.com:443/argentina
country_name : Armenia  data-url : https://taxsummaries.pwc.com:443/armenia
country_name : Australia  data-url : https://taxsummaries.pwc.com:443/australia
country_name : Austria  data-url : https://taxsummaries.pwc.com:443/austria
country_name : Azerbaijan  data-url : https://taxsummaries.pwc.com:443/azerbaijan
country_name : Bahrain  data-url : https://taxsummaries.pwc.com:443/bahrain
country_name : Barbados  data-url : https://taxsummaries.pwc.com:443/barbados
country_name : Belarus  data-url : https://taxsummaries.pwc.com:443/belarus
country_name : Belgium  data-url : https://taxsummaries.pwc.com:443/belgium
country_name : Bermuda  data-url : https://taxsummaries.pwc.com:443/bermuda
country_name : Bolivia  data-url : https://taxsummaries.pwc.com:443/bolivia
country_name : Bosnia and Herzegovina  data-url : https://taxsummaries.pwc.com:443/bosnia-and-herzegovina
country_name : Botswana  data-url : https://taxsummaries.pwc.com:443/botswana
country_name : Brazil  data-url : https://taxsummaries.pwc.com:443/brazil
country_name : Bulgaria  data-url : https://taxsummaries.pwc.com:443/bulgaria


and so on ......

For getting as a list:

for each in options:
    l.append( str(each.text)+ " : " + str(each['data-url']))
print(l)

Output will be:

['Albania : https://taxsummaries.pwc.com:443/albania', 'Algeria : https://taxsummaries.pwc.com:443/algeria', 'Angola : https://taxsummaries.pwc.com:443/angola', 'Argentina : https://taxsummaries.pwc.com:443/argentina', 'Armenia : https://taxsummaries.pwc.com:443/armenia', 'Australia : https://taxsummaries.pwc.com:443/australia', 'Austria : https://taxsummaries.pwc.com:443/austria', 'Azerbaijan : https://taxsummaries.pwc.com:443/azerbaijan', 'Bahrain : https://taxsummaries.pwc.com:443/bahrain', 'Barbados : https://taxsummaries.pwc.com:443/barbados', 'Belarus : https://taxsummaries.pwc.com:443/belarus', 'Belgium : https://taxsummaries.pwc.com:443/belgium', 'Bermuda : https://taxsummaries.pwc.com:443/bermuda', 'Bolivia : https://taxsummaries.pwc.com:443/bolivia', 'Bosnia and Herzegovina : https://taxsummaries.pwc.com:443/bosnia-and-herzegovina', 'Botswana : https://taxsummaries.pwc.com:443/botswana', 'Brazil : https://taxsummaries.pwc.com:443/brazil', 'Bulgaria : https://taxsummaries.pwc.com:443/bulgaria', 'Cabo Verde : https://taxsummaries.pwc.com:443/cabo-verde', 'Cambodia : https://taxsummaries.pwc.com:443/cambodia', 'Cameroon, Republic of : https://taxsummaries.pwc.com:443/republic-of-cameroon',


and so on............]

Python -- Beautiful Soup -- Returning Information if a tag is empty or has values

2 Answers