Extracting data from Airtasker using beautiful Soup

user_13408417 0
Viewed 46

I am trying to extract data from this website - https://www.airtasker.com/users/brad-n-11346775/.

So far, I have managed to extract everything except the license number. The problem I'm facing is bizarre as the license number is in the form of text. I was able to extract everything else like the Name, Address etc. For example, to extract the Name, I just did this:

name.append(pro.find('div', class_= 'name').text)

And it works just fine.

This is what I have tried to do, but I'm getting the output as None

license_number.append(pro.find('div', class_= 'sub-text'))

When I do :

license_number.append(pro.find('div', class_= 'sub-text').text)

It gives me the following error:

AttributeError: 'NoneType' object has no attribute 'text'

That means it does not recognise the license number as a text, even though it is a text.

Can someone please give me a workable solution and please tell me what am I doing wrong??? Regards,

beautifulsoup html python
1 Answers

The badge with the license number is added to the HTML dynamically from a Boostrap JSON that sits in one of the <script> tags.

You can find the tag with bs4 and scoop out the data with regex and parse it with json.

Here's how:

import ast
import json
import re

import requests
from bs4 import BeautifulSoup

page = requests.get("https://www.airtasker.com/users/brad-n-11346775/").text
scripts = BeautifulSoup(page, "lxml").find_all("script")[-4]
bootstrap_JSON = json.loads(
    ast.literal_eval(re.search(r"parse\((.*)\)", scripts.string).group(1))
)
print(bootstrap_JSON["profile"]["badges"]["electrical_vic"]["reference_code"])

Output:

Licence No. 28661
user_61067910
Related