BeautifulSoup, a dictionary from an HTML table

user_1145627 0
Viewed 31598

I am trying to scrape table data from a website.

Here is a simple example table:

t = '<html><table>' +\
    '<tr><td class="label"> a </td> <td> 1 </td></tr>' +\
    '<tr><td class="label"> b </td> <td> 2 </td></tr>' +\
    '<tr><td class="label"> c </td> <td> 3 </td></tr>' +\
    '<tr><td class="label"> d </td> <td> 4 </td></tr>' +\
    '</table></html>'

Desired parse result is {' a ': ' 1 ', ' b ': ' 2 ', ' c ': ' 3 ', ' d ' : ' 4' }

This is my closest attempt so far:

for tr in s.findAll('tr'):
  k, v = BeautifulSoup(str(tr)).findAll('td')
  d[str(k)] = str(v)

Result is:

{'<td class="label"> a </td>': '<td> 1 </td>', '<td class="label"> d </td>': '<td> 4 </td>', '<td class="label"> b </td>': '<td> 2 </td>', '<td class="label"> c </td>': '<td> 3 </td>'}

I'm aware of the text=True parameter of findAll() but I'm not getting the expected results when I use it.

I'm using python 2.6 and BeautifulSoup3.

beautifulsoup python
5 Answers

BeautifulSoup and Python have evolved, so if someone comes here with newer versions:

Python>=3.7
BeautifulSoup>=4.7

Here's updated code that works:

# import bs4 and create your 'soup' object

table = soup.find('table')

headers = [header.text for header in table.find_all('th')]
results = [{headers[i]: cell for i, cell in enumerate(row.find_all('td'))}
           for row in table.find_all('tr')]
user_57175800

Here is a solution that I believe is closer to the original attempt:

d={}
for tr in soup.findAll('tr'):
    key = tr.text.split()[0]
    val = tr.text.split()[1]
    d[key] = val

print(d)
user_147764270
Related