Python beautifulsoup - get all text separated by break tag

user_5134203 0
Viewed 274

I have the following tables:

<table width="100%" border="0" cellspacing="2" cellpadding="0">
                      <tbody><tr> 
                        <td class="labelplain">ANGARA, EDGARDO J.<br>ENRILE, JUAN PONCE<br>MAGSAYSAY JR., RAMON B.<br>ROXAS, MAR<br>GORDON, RICHARD "DICK" J.<br>FLAVIER, JUAN M.<br>MADRIGAL, M. A.<br>ARROYO, JOKER P.<br>RECTO, RALPH G.<br></td>
                      </tr>
                    </tbody></table>

I can traverse towards this part of the HTML using the following code below:

soup.find('td', text = re.compile('Co-author\(s'), attrs={'class': 'labelplain'}).find_next('td')
coauthor = soup.find('td', text = re.compile('Co-author\(s'), attrs={'class': 'labelplain'}).find_next('td')

I am able to get the text using the one below:

for br in coauthor.find_all('br'):
  firstcoauthor = (br.previousSibling)
  print (firstcoauthor)

The output I want to arrive to is a result of all text then separated by a semicolon (;) like so: ANGARA, EDGARDO J.;ENRILE, JUAN PONCE;MAGSAYSAY JR., RAMON B.;ROXAS, MAR;GORDON, RICHARD "DICK" J.;FLAVIER, JUAN M.;MADRIGAL, M. A.;ARROYO, JOKER P.;RECTO, RALPH G.

But the code above gives me a result like below:

ANGARA, EDGARDO J.
ENRILE, JUAN PONCE
MAGSAYSAY JR., RAMON B.
ROXAS, MAR
GORDON, RICHARD "DICK" J.
FLAVIER, JUAN M.
MADRIGAL, M. A.
ARROYO, JOKER P.
RECTO, RALPH G.

I've tried the replace function but to no avail.

print (firstcoauthor.replace("\n", ";"))

and

print (firstcoauthor.replace("\r\n", ";"))

Even escape \r\n and \n like so:

print (firstcoauthor.replace("\\n", ";"))

How do I address my use case?

web-scraping beautifulsoup python replace string
2 Answers

Think it is much more simpler to get that result with setting join/delimiter parameter to get_text():

soup.find('td').get_text(';')

Based on your example you will get:

ANGARA, EDGARDO J.;ENRILE, JUAN PONCE;MAGSAYSAY JR., RAMON B.;ROXAS, MAR;GORDON, RICHARD "DICK" J.;FLAVIER, JUAN M.;MADRIGAL, M. A.;ARROYO, JOKER P.;RECTO, RALPH G.

EDIT

Based on the behaviour, extra semicolons, mentioned in your comment, I suspect that the structure of the element is different from the one in the question and has extra breaks.

In that case, I would change the strategy and recommend to:

  • add additional strip parameter to get_text():

    soup.find('td').get_text(';', strip=True)

  • or use a join() from stripped_strings, what is doing almost the same:

    ';'.join(soup.find('td').stripped_strings)
Example HTML

Added additional <br>, spaces and linebreaks to the HTML.

html = '''
<table width="100%" border="0" cellspacing="2" cellpadding="0">
    <tbody><tr>
    
    <br>
           <td class="labelplain">
           ANGARA, EDGARDO J.<br>ENRILE, JUAN PONCE<br>MAGSAYSAY JR., RAMON B.<br>ROXAS, MAR<br>GORDON, RICHARD "DICK" J.<br>FLAVIER, JUAN M.<br>MADRIGAL, M. A.<br>ARROYO, JOKER P.<br>RECTO, RALPH G.<br> 
           
           <br>
           </td>
           </tr>
</tbody></table>'''
Output
ANGARA, EDGARDO J.;ENRILE, JUAN PONCE;MAGSAYSAY JR., RAMON B.;ROXAS, MAR;GORDON, RICHARD "DICK" J.;FLAVIER, JUAN M.;MADRIGAL, M. A.;ARROYO, JOKER P.;RECTO, RALPH G.
user_144608240

Select the element you want - I'm just picking the td here... (but use whatever so you have the element with the embedded br elements)

data = soup.select_one('td')

Then... replace all the br elements with a semi-colon:

for br in data.select('br'):
    br.replace_with(';')

Get your element's text:

output = data.get_text()
# 'ANGARA, EDGARDO J.;ENRILE, JUAN PONCE;MAGSAYSAY JR., RAMON B.;ROXAS, MAR;GORDON, RICHARD "DICK" J.;FLAVIER, JUAN M.;MADRIGAL, M. A.;ARROYO, JOKER P.;RECTO, RALPH G.;'
user_12527590
Related