BeautifulSoup - finding a tag with a certain attribute value doesn't work well

user_10820100 0
Viewed 38

I am new to python crawling. I played with find(tag, attribute = 'value') and found some interesting results.

Here is the code:

from bs4 import BeautifulSoup

html = '''
<a class="cluster_text_headline nclicks(cls_pol.clsart)" href="https://news.naver.com/main/read.nhn?mode=LSD&amp;mid=shm&amp;sid1=100&amp;oid=421&amp;aid=0004679941">text1</a>
<a class="cluster_text_headline nclicks(cls_pol.clsart)" href="https://news.naver.com/main/read.nhn?mode=LSD&amp;mid=shm&amp;sid1=100&amp;oid=032&amp;aid=0003013328">text2</a>
<a class="cluster_text_headline nclicks(cls_pol.clsart)" href="https://news.naver.com/main/read.nhn?mode=LSD&amp;mid=shm&amp;sid1=100&amp;oid=052&amp;aid=0001449034">text3</a>
<a class="cluster_text_headline nclicks(cls_pol.clsart)" href="https://news.naver.com/main/read.nhn?mode=LSD&amp;mid=shm&amp;sid1=100&amp;oid=055&amp;aid=0000819508">text4</a>
<a class="cluster_text_headline nclicks(cls_pol.clsart)" href="https://www.test.com">text5</a>

'''
soup = BeautifulSoup(html, 'html.parser')
a_tags = soup.find('a', href="https://www.test.com")
print(a_tags)

This returns <a class="cluster_text_headline nclicks(cls_pol.clsart)" href="https://www.test.com">text5</a> as expected.

But when I replaced the attribute value in .find(), for example, with 'https://news.naver.com/main/read.nhn?mode=LSD&amp;mid=shm&amp;sid1=100&amp;oid=032&amp;aid=0003013328' (the second link in the code), None is returned. It would be appreciated if anyone can explain why outcomes differ depending on the URL.

beautifulsoup attributes python
1 Answers

this is caused by the html parser bs4 uses, and the &amp; in your hrefs: when it parses your html source, 

soup = BeautifulSoup(html, 'html.parser')

it unescapes &amp;, converting &amp; into &. (see: https://github.com/python/cpython/blob/a5d6aba318ead9cc756ba750a70da41f5def3f8f/Lib/html/parser.py#L326).

it apparently does not do the same conversion with the href value you pass to find

a_tags = soup.find('a', href='https://news.naver.com/main/read.nhn?mode=LSD&amp;mid=shm&amp;sid1=100&amp;oid=032&amp;aid=0003013328')

so it ends up trying to match your (escaped) href value https://news.naver.com/main/read.nhn?mode=LSD&amp;mid=shm&amp;sid1=100&amp;oid=032&amp;aid=0003013328

against the unescaped href attribute in your source <a href="https://news.naver.com/main/read.nhn?mode=LSD&mid=shm&sid1=100&oid=032&aid=0003013328">text2</a>

...and fails. this on the other hand, returns the "expected" result:

html = '''
<a href="https://news.naver.com/main/read.nhn?mode=LSD&amp;mid=shm&amp;sid1=100&amp;oid=032&amp;aid=0003013328">text2</a>
<a class="cluster_text_headline nclicks(cls_pol.clsart)" href="https://www.test.com">text5</a>
'''

if __name__ == "__main__":
    soup = BeautifulSoup(html, 'html.parser')
    a_tags = soup.find('a', href='https://news.naver.com/main/read.nhn?mode=LSD&mid=shm&sid1=100&oid=032&aid=0003013328')
    print(a_tags)
    a_tags = soup.find('a', href='https://www.test.com')
    print(a_tags)
user_126937280
Related