BeautifulSoup: split tag (containing other tags) into two at string

user_2663568 0
Viewed 256

I’m massaging some HTML dictionary data into XML for import into some dictionary software.

The original HTML looks something like this:

<div class="entry">
  <span class="headword">word</span> 
  <span class="pos">part of speech</span> 
  <span class="definition">sense1; sense2 
    <span class="example">(example2.1; example2.2)</span>
    ; sense3 <span class="example">(example3.1; example3.2)</span>
  </span> 
</div>

Edit: In fact, the input‘s classes don’t neatly match the output XML tags. That was just to make the relationship clear in my example. I need to replace specific classes with specific XML tags, but they don’t match.

The ideal end result would look like this:

<entry>
  <headword>word</headword>
  <pos>part of speech</pos>
  <sense>
    <definition>sense1</definition>
  </sense>
  <sense>
    <definition>sense2</definition>
    <example>example2.1</example>
    <example>example2.2</example>
  </sense>
  <sense>
    <definition>sense3</definition>
    <example>example3.1</example>
    <example>example3.2</example>
  </sense>
</entry>

The current state of my soup (having done the straightforward replacements) is:

<entry>
  <headword>word</headword>
  <pos>part of speech</pos>
  <definition>sense1; sense2
    <example>example2.1</example>
    <example>example2.2</example>
    ; sense3 
    <example>example3.1</example>
    <example>example3.2</example>
  </definition>
</entry>

The divisions that map 1:1 are easy and wrapping the definitions+examples in a sense tag should be too, but the problem is the way the original indiscriminately uses ; to separate both senses and examples. This means I need to split the example tags first and then the definition tags at ; (i.e. effectively replace ; with </example>\n<example> or </definition>\n<definition>). Since I started writing this question I’ve figured out how to do that for the examples (since they only contain strings), but the definitions are very likely to contain <example> tags themselves, so I can’t just use split() because a list is returned & 'list' object has no attribute 'split'.

Is there any easier way to split tags containing other tags or do I have to iterate through the resultset list and recreate all the tags?

tags = soup.find_all("example")
for tag in tags:
    tag.string = re.sub(r"[()]", "", tag.string)     # remove parentheses
    egs = tag.string.split("; ")     # or str(tag.contents).split("; ") ?
    new = ""
    if len(egs) > 1:
        for eg in reversed(egs[1:]):
            new = soup.new_tag("example")
            new.string = eg
            tag.insert_after(new)
        tag.string = egs[0]             # orig tag becomes 1st seg only
beautifulsoup xml html python
1 Answers

You can inspect soup.contents for each element and build the structure by recursively traversing the non-string elements in soup.contents:

from bs4 import BeautifulSoup, NavigableString
import re
def to_xml(d):
   r, s, k = [], None, []
   for i in filter(lambda x:x != '\n', d.contents):
      if isinstance(i, NavigableString):
         if s is not None:
            r.append((s, k))
         s = [j for i in re.sub('^\(|\)$', '', i).split('; ') if (j:=re.sub('^\W+|\W+$', '', i))]
         k = []
      else:
         k.append(i)
   r.append((s, k))
   for a, b in r:
      if a is not None:
         if len(a) == 1 and not b:
            yield f'<{(c:=" ".join(d["class"]))}>{a[0]}</{c}>\n'
         elif not b:
            yield from ["<{}>\n<{}>{}</{}>\n</{}>\n".format(c, c1, i, c1, c) if (c:=re.sub('[\d+\.]+$', '', i)) != (c1:=" ".join(d["class"])) else f"<{c}>{i}</{c}>" for i in a]
         else:
            yield from ["<{}>\n<{}>{}</{}>\n</{}>\n".format((c:=re.sub('[\d+\.]+$', '', i)), (c1:=" ".join(d["class"])), i, c1, c) for i in a[:-1]]
            yield "<{}>\n<{}>{}</{}>\n{}\n</{}>\n".format((c:=re.sub('[\d+\.]+$', '', a[-1])), (c1:=' '.join(d['class'])), a[-1], c1, '\n'.join(j for k in b for j in to_xml(k)), c)
      else:
          yield '<{}>{}</{}>'.format((c1:=" ".join(d["class"])), "\n".join(j for k in b for j in to_xml(k)), c1)
        

s = """
 <div class="entry">
   <span class="headword">word</span> 
   <span class="pos">part of speech</span> 
   <span class="definition">sense1; sense2 
   <span class="example">(example2.1; example2.2)</span>
    ; sense3 <span class="example">(example3.1; example3.2)</span>
   </span> 
 </div>
"""
r = BeautifulSoup(''.join(to_xml(BeautifulSoup(s, 'html.parser').div)), 'html.parser')
print(r)

Output:

<entry>
   <headword>word</headword>
   <pos>part of speech</pos>
   <sense>
      <definition>sense1</definition>
   </sense>
   <sense>
      <definition>sense2</definition>
      <example>example2.1</example>
      <example>example2.2</example>
   </sense>
   <sense>
      <definition>sense3</definition>
      <example>example3.1</example>
      <example>example3.2</example>
   </sense>
</entry>
user_73267380
Related