Extract text in proper format (with spaces in between) from <td> tags using beautiful soup

user_10419999 0
Viewed 471

I am trying to extract column headings from one of the tables from ABBV 10-k sec filing (`Issuer Purchases of Equity Securities' table on page 25 - below the graph.)

inside <td> tag in the column heading <tr> tag, text is in separate <div> tags as in the example below

<tr>
  <td>
    <div>string1</div>
    <div>string2</div>
    <div>string3</div>
  </td>
</tr>

when trying to extract all text fro a tag, there is no space separation between texts (e.g. for the above html output will be string1string3string3 expectedstring1 string3 string3).

Using below code to extract column headings from table

url = 'https://www.sec.gov/Archives/edgar/data/1551152/000155115218000014/abbv-20171231x10k.htm'
htmlpage = requests.get(url)
soup = BeautifulSoup(htmlpage.text, "lxml")
table = soup.find_all('table')[76]
rows = table.find_all('tr')
table_data = []
for tr in rows[2:3]:
    row_data=[]
    cells = tr.find_all(['td', 'th'], recursive=False)
    for cell in cells[1:4]:
        row_data.append(cell.text.encode('utf-8'))
    table_data.append([x.decode('utf-8').strip() for x in row_data])

print(table_data)

output:[['(a) TotalNumberof Shares(or Units)Purchased', '', '(b) AveragePricePaid per Share(or Unit)']]

Expected output:[['(a) Total Number of Shares (or Units) Purchased', '', '(b) Average Price Paid per Share (or Unit)']] (each word separated bay a space)

beautifulsoup python-3.x html python
1 Answers

use the separator parameter with .get_text():

html = '''<tr>
  <td>
    <div>string1</div>
    <div>string2</div>
    <div>string3</div>
  </td>
</tr>'''

import bs4

soup = bs4.BeautifulSoup(html, 'html.parser')

td = soup.find('td')
td.get_text(separator=' ')

Here's how it looks with your code:

from bs4 import BeautifulSoup
import requests

url = 'https://www.sec.gov/Archives/edgar/data/1551152/000155115218000014/abbv-20171231x10k.htm'
htmlpage = requests.get(url)
soup = BeautifulSoup(htmlpage.text, "lxml")
table = soup.find_all('table')[76]
rows = table.find_all('tr')
table_data = []
for tr in rows[2:3]:
    row_data=[]
    cells = tr.find_all(['td', 'th'], recursive=False)
    for cell in cells[1:4]:
        row_data.append(cell.get_text(separator=' ').encode('utf-8'))
    table_data.append([x.decode('utf-8').strip() for x in row_data])

print(table_data)

Output:

print(table_data)
[['(a) Total Number of Shares (or Units) Purchased', '', '(b) Average Price Paid per Share (or Unit)']]
user_77522960
Related