I don't understand why Beautiful Soup gives first link not second, and first select_one won't work as second one does

user_16061996 0
Viewed 34

I am doing some Python coding with Beautiful Soup. I got it working now, but I have no idea why it works, and I am pretty sure this will cause me to run into problems as I go forward. Can anyone help me understand this?

I have downloaded an HTML file that has hundreds of items in a table. Here is one table item:

<td class="gsc_a_t">
  <a href="www.the_first_link.com" class="gsc_a_at">The title that goes with the link</a>
  <div class="gs_gray">author1, author 2, author 3</div>
  <div class="gs_gray">
    ”source for the item”
    <span class="gs_oph">, 2020</span>
  </div>
</td>
<td class="gsc_a_c">
  <a href="www.the_second_link.com” class="gsc_a_ac gs_ibl">2</a>
</td>
<td class="gsc_a_y">
  <span class="gsc_a_h gsc_a_hc gs_ibl">2020</span>
</td>

The following is the code for each table item:

for i in table_results:
    item = i
    first_link = i.a['href']
    title = i.a.text
    authors = i.select_one('.gs_gray').text
    source = i.select('.gs_gray')[-1].text
    second_link = i.select_one('.gsc_a_ac')['href']
    citations = i.select_one('.gsc_a_ac').text
    year = i.select_one('.gsc_a_y').text

When I run it, I want first_link to record the first link and second_link to get the second one. They ARE doing that ... but I don't know why. Why doesn't the first link capture the i.a['href'] from the second link instead of the first link? Both have the same <a href=" ... ></a> constructions. Also, I tried doing first_link = i.select_one('.gsc.a.at')['href'] similar to the way it captures the second link, and that does not work. It references "return self.attrs[key]" and says KeyError: 'href'

I have looked at the Beautiful Soup documentation, but I am not understanding this.

beautifulsoup html python
1 Answers

Why doesn't the first link capture the i.a['href'] from the second link instead of the first link?

I don't have the reference handy, but using dot notation selects the first element with the particular name. It's the equivalent of using select_one with css selectors. Which leads us to:

first_link = i.select_one('.gsc.a.at')['href'] ... says KeyError: 'href'

That's because select_one selects the first element meeting the stated condition (in this case, having the specified class attribute value). In the html, the first element meeting that condition is the first td, which itself doesn't have a href attribute. What you are looking for is the child a element of that td:

i.select_one('.gsc_a_t a')['href']
user_94480900
Related