BeautifulSoup Parse Text after <b> and before </br>

Viewed 524

I have this code trying to parse search results from a grant website (please find the URL in the code, I can't post the link yet until my rep is higher), the "Year"and "Amount Award" after tags and before tags.

enter image description here

Two questions:

1) Why is this only returning the 1st table?

2) Any way I can get the text that is after the (i.e. Year and Amount Award strings) and (i.e. the actual number such as 2015 and $100000)

Specifically:

<td valign="top">
                <b>Year: </b>2014<br>
                <b>Award Amount: </b>$84,907                                                                                    </td>

Here is my script:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'http://www.ned.org/wp-content/themes/ned/search/grant-search.php?' \
    'organizationName=&region=ASIA&projectCountry=China&amount=&fromDate=&toDate=&' \
    'projectFocus%5B%5D=&search=&maxCount=25&orderBy=Year&start=1&sbmt=1'

r = requests.get(url)

html_content = r.text

soup = BeautifulSoup(html_content, "html.parser")

tables = soup.find_all('table')


data = {
        'col_names': [],
        'info' : [],
        'year_amount':[]
        }


index = 0

for table in tables:
    rows = table.find_all('tr')[1:]
    for row in rows:
        cols = row.find_all('td')
        data['col_names'].append(cols[0].get_text())
        data['info'].append(cols[1].get_text())
        try:
            data['year_amount'].append(cols[2].get_text())
        except IndexError:
            data['year_amount'].append(None)
    grant_df = pd.DataFrame(data)
    index += 1
    filename = 'grant ' + str(index) + '.csv'
    grant_df.to_csv(filename)
1 Answers
Related