KeyError: -1 when appending new tag to soup in bs4

Question

I have a template htm file with a table in it, which has one column and looks like this:


 
   ROWNAME1
 
 
   ROWNAME2

I'm trying to create new td tags by iterating over the tr tags in the template and appending a new td inside every tr tag, so it will look like this, if I iterate over one of my input files:


 
   ROWNAME1
   ROWNAME1
 
 
   ROWNAME2
   ROWNAME2

So, I'm basically copying the first td tag in a tr and appending it to the original template html. This happens by iterating over the rows (tr tags) of my template html. At each row, I fill the contents of a newly created td with the contents from a separate input file, which has a corresponding td tag with new information in it. The idea is to take a column from an input file and append it to the table in the template html. Each file I iterate over adds a new column to the table in the template.

This works fine for the first file. I can append the first new column to my template html no problem. But when I try to add the next column, the script throws the following error:

File "C:\Users\Chris\Desktop\Nothing\Henke Skripte\HTML_Tables_transform\Run	emp\Parameter_Tables.py", line 124, in content_replacer
    parent_tag.append(new_td)
  File "C:\Python\Python38\lib\site-packages\bs4\element.py", line 351, in append
    self.insert(len(self.contents), tag)
  File "C:\Python\Python38\lib\site-packages\bs4\element.py", line 315, in insert
    new_child.previous_element = previous_child._last_descendant(False)
  File "C:\Python\Python38\lib\site-packages\bs4\element.py", line 268, in _last_descendant
    last_child = last_child.contents[-1]
  File "C:\Python\Python38\lib\site-packages\bs4\element.py", line 992, in __getitem__
    return self.attrs[key]
KeyError: -1

Apparently, appending the new td works over 20 times, when I do it for the first column/input file. But directly when the next input file is used and I try to add the second column, a bs4 function tries to access the attrs dictionary of the newly appended td with the key "-1", which does not exist and I don't know why it would think that the key -1 should exist in the first place.

Here's my shortened code with an additional print of the td that causes the problem:

def content_replacer(parent_tag,content,parent_html):

    '''Takes a td, replaces the content appropriately and returns it.'''

    html_content = BeautifulSoup(content,features='html.parser')

    # Go down the hierachy of children of td to create new td with the full formatting.
    td_tag = parent_tag.select('td')
    if len(td_tag) > 1:
        print(td_tag[0].prettify())
        print(td_tag[0].attrs)
    new_td = parent_html.new_tag('td', attrs=td_tag[0].attrs)
    parent_tag.append(new_td)


    p_tag = parent_tag.select('p')
    new_p = parent_html.new_tag('p', attrs=p_tag[0].attrs)
    new_td.append(new_p)

    span_tag = parent_tag.select('span')                    
    new_span = parent_html.new_tag('span', attrs=span_tag[0].attrs)
    new_p.append(new_span)

    new_span.contents = html_content



# mainloop over input files and tr tags.
all_input_files = glob('*.html')
htm_file = html_parser('template.htm')
nrows = len(htm_file.table.find_all('tr'))

row_name_cleaner = lambda tag: re.sub('[ ]{2,}',' ',tag.text.strip().replace('
',''))

for i in range(len(all_input_files)):
    # html_parser just reads in htmls and turns them to soup.
    input_file = html_parser(all_input_files[i])
    # content_dictionary creates a dict of input data to be input into the new td tags
    content_dict = content_dictionary(input_file)
    for j in range(nrows):
        # get current row (tr tag)
        htm_row = htm_file.table.select('tr')[j]
        # Get info on current row.
        row_name = row_name_cleaner(htm_row)
        # Translate row name into slicing string for content_dict
        slice_string = slice_string_translator(row_name)
        # Get new content from input file
        new_content = content_dict[slice_string]
        content_replacer(htm_row,new_content,htm_file)

The print of the faulty td that is supposed to be appended looks like this:


 
  
   Compound
   
   
  
 


{'width': '218', 'style': 'width:163.55pt;border-top:solid windowtext 1.0pt;
  border-left:none;border-bottom:solid windowtext 1.0pt;border-right:none;
  mso-border-top-alt:solid windowtext .5pt;mso-border-bottom-alt:solid windowtext .5pt;
  padding:0cm 5.4pt 0cm 5.4pt;height:11.45pt'}
Traceback (most recent call last):
  File "C:\Users\Chris\Desktop\Nothing\Henke Skripte\HTML_Tables_transform\Run	emp\Parameter_Tables.py", line 165, in 
    content_replacer(htm_row,new_content,htm_file)
  File "C:\Users\Chris\Desktop\Nothing\Henke Skripte\HTML_Tables_transform\Run	emp\Parameter_Tables.py", line 124, in content_replacer
    parent_tag.append(new_td)
  File "C:\Python\Python38\lib\site-packages\bs4\element.py", line 351, in append
    self.insert(len(self.contents), tag)
  File "C:\Python\Python38\lib\site-packages\bs4\element.py", line 315, in insert
    new_child.previous_element = previous_child._last_descendant(False)
  File "C:\Python\Python38\lib\site-packages\bs4\element.py", line 268, in _last_descendant
    last_child = last_child.contents[-1]
  File "C:\Python\Python38\lib\site-packages\bs4\element.py", line 992, in __getitem__
    return self.attrs[key]
KeyError: -1

Here's the full tr tag where the error occurs.



Compound


Compound
PK59zni_co_100K_red

user_2616577 · Answer

Let me go first for the easy fix and then for the reason of the cryptic message, which hopefully is illuminating.

Solution

In the line

new_span.contents = html_content

replace it with

new_span.contents = html_content.contents

Referring to the documentation, the contents attribute is a list of children. By assigning it something that is not a list, the internal library code breaks the next time you come around. That's why it works for the first file but not the second.

In fact, it might be better to use

new_span.append(html_content)

since it's more semantic and will take care of metadata. The library could take care of this with a setter but it doesn't seem to be the case.

Why the cryptic message?

The following uses the code for BeautifulSoup v.4.8.2, particularly the file element.py.

Calling the .append() function is the same as inserting at the end of the structure.

def append(self, tag):
  self.insert(position=len(self.contents), new_child=tag)

def insert(self, position, new_child):
  """Insert a new PageElement in the list of this PageElement's children.
     This works the same way as `list.insert`.
     :param position: The numeric position that should be occupied
        in `self.children` by the new PageElement. 
     :param new_child: A PageElement.
  """
  ...

Inside the insert() code, the previous_child object is, in our case, defined as the last structure in the contents list.

previous_child = self.contents[position - 1]

Remember new_child is the new tag we're inserting. The next line is key

new_child.previous_element = previous_child._last_descendant(False)

What this is doing is creating an attribute that refers to the last descendant of the previous child.

This can intuitively be seen for the following minimal example

tree = BeautifulSoup(
  'string1
string2string3',
  features='html.parser',
)
tree._last_descendant()  # 'string3'

Let us check then how _last_descendant is defined.

def _last_descendant(self, is_initialized=True, accept_self=True):
  """Finds the last element beneath this object to be parsed."""
  ...
  while isinstance(last_child, Tag) and last_child.contents:
    last_child = last_child.contents[-1]
  ...

So the algorithm will roughly proceed as follows

tree.contents
# [(Tag): string1
string2string3]
tree.contents[-1]
# (Tag): string1
string2string3
tree.contents[-1].contents
# [(Tag): string1
string2string3]
tree.contents[-1].contents[-1]
# (Tag): string1
string2string3
tree.contents[-1].contents[-1].contents
# [(Tag): string1
, (Tag): string2string3]
...
tree.contents[-1].contents[-1].contents[-1].contents[-1]
# 'string3'
# No longer an instance of Tag, but a NavigableString. Stop.

This assumes, rightfully so, that the contents attribute is always a list. The issue comes when it's not, getting you a KeyError.

KeyError: -1 when appending new tag to soup in bs4

1 Answers

Solution

Why the cryptic message?