Python BeautifulSoup how can I get data for the latest selector

user_3556956 0
Viewed 153

After sending a python HTTP request, it's response (data) has a html page which has many blocks of the ABCD . Here is one snippet 

                   <tr>
                        <td class="success"></td>
                        <td class="truncate">ABCD</td>
                        <td>12/18/2018 21:45</td>
                        <td>12/18/2018 21:46</td>
                        <td>10</td>
                        <td>10</td>
                        <td>100.0</td>
                        <td><span class="label success">Success</span></td>
                        <td>SMS</td>
                        <td>
                            <a data-id="134717" class="btn" title="Go">View</a>
                        </td>
                    </tr>

I need to retrieve the most recent data-id for ABCD (in this case 134717, and this number is dynamic). Also note there are many of those ABCD's with different dates, I want the most recent .

I can do it using regular expression and going through line by line. But I think it is better to do it with with BeautifulSoup.

I tried this it finds all ABCDs but I dont know how to get the most recent one :

    soup = BeautifulSoup(data, "html.parser")
    for i in soup.select("td.truncate"):
        #print(i.text)
        if i.text == "ABCD":
            print ("Got it ", i.text)
            id1 = soup.select_one("a.data-id")
            print (id1)
            parsed_url1 = urlparse(id1)
beautifulsoup python
3 Answers

You'll need the dateutils parser for this one. Obviously there's no way of telling which <td> has the date in it, so you just have to iterate over all the td's in the matched tr's, and try to parse the datetime, and if the datetime parsing was a success just append it to the dates list for a specific id. After you've gained all the dates for each ID, you just max on them to find the latest. 

from dateutil import parser as du_parser    
from collections import defaultdict
from bs4 import BeautifulSoup as BS

data = "<tr><td class=\"success\"></td><td class=\"truncate\">ABCD</td><td>12/18/2018 21:45</td><td>12/18/2018 21:46</td><td>10</td><td>10</td><td>100.0</td><td><span class=\"label success\">Success</span></td><td>SMS</td><td><a data-id=\"134717\" class=\"btn\" title=\"Go\">View</a></td></tr>"
b1 = BS(data, "html.parser")

td_of_interest = b1.find_all("td")
tr_that_contain_our_td = [x.parent for x in b1.find_all("td", string="ABCD")]

ids_dict = defaultdict(list)

# iterate over matched tr's to get their dates
for tr in tr_that_contain_our_td:
    extracted_id = tr.find("a")['data-id']

    for td in tr.find_all("td"):
        try:
            if len(td.contents) > 0:
                actual_date = du_parser.parse(td.contents[0])
                ids_dict[extracted_id].append(actual_date)
        except ValueError:
            pass  #nothing to do here

ids_dict = {k: max(v) for k, v in ids_dict.items()}

print(ids_dict)
user_8585650

assuming html follows the same pattern:

given:

html = '''                   <tr>
                        <td class="success"></td>
                        <td class="truncate">ABCD</td>
                        <td>12/18/2018 21:45</td>
                        <td>12/18/2018 21:46</td>
                        <td>10</td>
                        <td>10</td>
                        <td>100.0</td>
                        <td><span class="label success">Success</span></td>
                        <td>SMS</td>
                        <td>
                            <a data-id="134717" class="btn" title="Go">View</a>
                        </td>
                    </tr>


                    <tr>
                        <td class="success"></td>
                        <td class="truncate">ABCD</td>
                        <td>12/20/2018 21:45</td>
                        <td>12/20/2018 21:46</td>
                        <td>99</td>
                        <td>99</td>
                        <td>999.0</td>
                        <td><span class="label success999">Success</span></td>
                        <td>SMS99</td>
                        <td>
                            <a data-id="9913471799" class="btn" title="Go">View</a>
                        </td>
                    </tr>

                                        <tr>
                        <td class="success"></td>
                        <td class="truncate">ABCD</td>
                        <td>12/22/2018 21:45</td>
                        <td>12/22/2018 21:46</td>
                        <td>99</td>
                        <td>99</td>
                        <td>999.0</td>
                        <td><span class="label success999">Success</span></td>
                        <td>SMS99</td>
                        <td>
                            <a data-id="found the latest date" class="btn" title="Go">View</a>
                        </td>
                    </tr>

                                        <tr>
                        <td class="success"></td>
                        <td class="truncate">ABCD</td>
                        <td>12/21/2018 21:45</td>
                        <td>12/21/2018 21:46</td>
                        <td>99</td>
                        <td>99</td>
                        <td>999.0</td>
                        <td><span class="label success999">Success</span></td>
                        <td>SMS99</td>
                        <td>
                            <a data-id="9913471799" class="btn" title="Go">View</a>
                        </td>
                    </tr>'''

find the latest date: 

import bs4
import re
import datetime                

dates_list = []

soup = bs4.BeautifulSoup(html, 'html.parser')

for i in soup.select("td.truncate"):
        #print(i.parent.text)
        match = re.search(r'\d{2}/\d{2}/\d{4}', i.parent.text)
        date = datetime.datetime.strptime(match.group(), '%m/%d/%Y').date()
        date = date.strftime('%m/%d/%Y')
        dates_list.append(date)

dates_list.sort()        
most_recent = dates_list[-1]

rows = soup.find_all('tr')
for row in rows:
    if str(most_recent) in row.text:
        id1 = row.find("a").get('data-id')  
        print (id1)
user_77522960

If the data-id is increasing number you can select a tag with highest data-id value with max().

recentDataID = max([x.get('data-id') for x  in soup.select("a[data-id]")])
print(recentDataID)

# if you want to select the parent or `tr`
mostRecentRow = soup.select_one('a[data-id=%s]' % recentDataID).parent.parent
user_4582140
Related