Soup not locating proper div tag when searched by text

Question

This is condensed version of the actual html which has many more tags.

html = '''



    
        Indicate by checkmark whether the registrant is a shell company (as defined in Rule 12b-2 of the Exchange Act). Yes
    

    
        ¨
    

    
        No
    

    
        x
    





    
        There were
    

    
        33,012,179
    

    
        shares of common stock, $.01 par value per share, outstanding at
    

    
        July 26, 2017
    

    
        .
    



'''

I am attempting to locate a tag based of text. The text is a form of regex which is located all within a div tag.

month_pattern = r'((Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|(Nov|Dec)(?:ember)?)\s?(\d{1,2}\D?)\s?(19[7-9]\d|20\d{2}|\d{2}))'

word_pattern = r'(?=.*common)(?=.*outstanding[.,]?)(?=.*shares[.,]?)(?=.*stock[.,]?)'


pattern = word_pattern + '.*' + month_pattern

The above regex is slightly complicated, but it works when I test it as a stand-alone test on on the text within div.

With the soup code below, I'm expecting a return of a type of soup object whose parent is the first div tag, however I am getting an empty list.

soup = bs(html, 'html.parser')

elem =  soup(text=re.compile(pattern, flags = re.IGNORECASE|re.DOTALL))
print(elem)

results in

[]

I suspect this problem is because the div's text is further nested within text? However, if I execute div.text all of the text is printed out, so I'm not sure why I am not getting any hits.

'''There were
    

        33,012,179
    

        shares of common stock, $.01 par value per share, outstanding at
    

        July 26, 2017
    

        .

        '''

Once again, regex is not a problem as via re module, I have:

print(re.search(pattern,text, flags = re.IGNORECASE|re.DOTALL))

with result:

<_sre.SRE_Match object; span=(0, 142), match='There were
    

        33,012,179
    

 >

Excepted Result:

I am expecting elem to be a non-empty list, so that if I run elem.parent as show here in accepted answer,

Using BeautifulSoup to find a HTML tag that contains certain text

I will be able to extract the first div tag with its inner html as follows:

  
    
        
            Indicate by checkmark whether the registrant is a shell company (as defined in Rule 12b-2 of the Exchange Act). Yes
        
    
        
            ¨
        
    
        
            No
        
    
        
            x

However, I am getting back an empty list, so elem.parent returns nothing if I iterate

Thank you.

Here is the full code for easy c&p:

#testing_html


from bs4 import BeautifulSoup as bs
import re
import os


html = '''



    
        Indicate by checkmark whether the registrant is a shell company (as defined in Rule 12b-2 of the Exchange Act). Yes
    

    
        ¨
    

    
        No
    

    
        x
    





    
        There were
    

    
        33,012,179
    

    
        shares of common stock, $.01 par value per share, outstanding at
    

    
        July 26, 2017
    

    
        .
    



'''

text = '''There were
    

        33,012,179
    

        shares of common stock, $.01 par value per share, outstanding at
    

        July 26, 2017
    

        .

        '''


month_pattern = r'((Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|(Nov|Dec)(?:ember)?)\s?(\d{1,2}\D?)\s?(19[7-9]\d|20\d{2}|\d{2}))'

word_pattern = r'(?=.*common)(?=.*outstanding[.,]?)(?=.*shares[.,]?)(?=.*stock[.,]?)'


pattern = word_pattern + '.*' + month_pattern

soup = bs(html, 'html.parser')

elem =  soup(text=re.compile(pattern, flags = re.IGNORECASE|re.DOTALL))
print(elem)

print(re.search(pattern,text, flags = re.IGNORECASE|re.DOTALL))

user_9448090 · Answer

I think I understand the problem now...

One issue you have is that your final regex expression pattern = word_pattern + '.*' + month_pattern can not find the target text because the target text is spread between several nodes so that no single node has the full pattern. In this case, the text is spread between two nodes. Both these nodes do have the same common grandfather - the

in question. You can get to it by calling parent twice.

This can be resolved with something along these lines:

elem_m =  soup(text=re.compile(month_pattern))
elem_w =  soup(text=re.compile(word_pattern))

if elem_m[0].parent.parent==elem_w[0].parent.parent:
    print((elem_m[0].parent.parent).text.strip())

More fundamentally, if you search around you'll see that using regex in the context of html/xml is highly discouraged. In order to avoid that, I would do something like this:

key_words = ['common','shares','stock,',"outstanding"]
months = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December'] 

for s in soup.select('*'):   
    words = all(word in s.text  for word in key_words)
    month = any(month in s.text  for month in months)
    if words == True and month == True:
        print(s.text.strip())

The output, in both cases, is:

There were
   

   33,012,179
   

   shares of common stock, $.01 par value per share, outstanding at
   

   July 26, 2017
   
        .

Good luck parsing EDGAR filings; not the most fun activity I can think of...

Soup not locating proper div tag when searched by text

Excepted Result:

1 Answers