My data from <script> returns None: AttributeError: 'NoneType' object has no attribute 'group'

user_8258990 0
Viewed 207

I try to get the data from within the <script> tag but 'model_data' return None. When I run the code I get the error:
model_data = model_data.group(1) AttributeError: 'NoneType' object has no attribute 'group'

What is wrong here?

html_doc = """
    <script>
      var modelData = {
        "hlsUrl": "null",
        "account": "4LH7J44IYPAGEZEY6E3UL"
      }
  </script>
"""

soup = BeautifulSoup(html_doc, "html.parser")
# locate the script, get the contents
script_text = soup.select_one("script").contents[0]

# get javascript object inside the script
model_data = re.search(r"modelData = ({.*?});", script_text, flags=re.S)
print(model_data) # RETURNS None - why?
model_data = model_data.group(1)

# "convert" the javascript object to json-valid object
model_data = re.sub(
    r"^\s*([^:\s]+):", r'"\1":', model_data.replace("'", '"'), flags=re.M
)

# json decode the object
model_data = json.loads(model_data)

# print the data
print(model_data["account"])

Updated issue:

After accepted the answer which worked with the given response, I found out that I had left out an important piece of information.

The full response is like this:

{
        "hlsUrl": "null",
        "account": "1V2FO4K7ME78RV09VXNEC",
        "packageName": "null",
        isActive: false
      }

Here it shows that isActive is not a json like object or what it's called, so it gives me now the following error:

obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 5 column 9 (char 111)
beautifulsoup python-3.x python
2 Answers

You're making it a bit too complicated than it really has to be.

Use .string not .contents[0]. As the later works too, the .string method doesn't need indexing and can be directly passed to re.search(). Also, IMHO, it's more readable.

Fix your regex.

This works:

modelData = ({.*?})

while this does not:

modelData = ({.*?});

Notice, there's no need for ;

Finally, you don't have to do all this:

# "convert" the javascript object to json-valid object
model_data = re.sub(
    r"^\s*([^:\s]+):", r'"\1":', model_data.replace("'", '"'), flags=re.M
)

Just simply dump the regex group(1) to json.loads.

Full code:

import json
import re

from bs4 import BeautifulSoup

html_doc = """
    <script>
      var modelData = {
        "hlsUrl": "null",
        "account": "4LH7J44IYPAGEZEY6E3UL"
      }
  </script>
"""

soup = BeautifulSoup(html_doc, "html.parser")
script_text = soup.select_one("script").string
model_data = re.search(r"modelData = ({.*?})", script_text, re.S).group(1)
print(json.loads(model_data)["account"])

Output:

4LH7J44IYPAGEZEY6E3UL
user_61067910

This looks like a case where you can search the HTML content for exactly what you're looking for instead of doing any intermediate parsing

html_doc = """
    <script>
      var modelData = {
        "hlsUrl": "null",
        "account": "4LH7J44IYPAGEZEY6E3UL"
      }
  </script>
"""

import re
next(re.finditer(r'"account":\s*"([\dA-Z]{21})"', html_doc)).group(1)
# '4LH7J44IYPAGEZEY6E3UL'

This works by finding the first instance of literally "account": followed by any amount of whitespace \s* and then a quoted string with exactly 21 uppercase digits or letters (tune 21 to the possible range)

re.finditer() and next() are helpful here because they'll

  • search the entire string until a match is found
  • get only the first instance or raise StopIteration
  • be surprisingly fast

While this avoids any intermediate parsing and doesn't attempt to parse the html with regex (which can only lead to ZA̡͊͠͝LGΌ), beware it can break if a similar field appears in unrelated scopes! If you expect such a case, adapting @baduker's answer will likely be more robust. Conversely, if you expect the structure to change, but not find an exactly-matching field ("account":\s*"([\dA-Z]{21})"), this may be more robust (as it doesn't care about the other HTML structure such as finding the <script>)

user_45410450
Related