How to get the first related tags with BeautifulSoup, xpath or css selectors

user_7521238 0
Viewed 122
<main>
   <span>
     <div id="1" class="infocard-list">
       <span>
         <div id="3" class="infocard-list">
         </div>
       </span>
       <span>
         <div id="4" class="infocard-list">
         </div>
       </span>
    </div>
    <div id="2" class="infocard-list">
       <span>
         <div id="5" class="infocard-list">
         </div>
       </span>
       <span>
         <div id="6" class="infocard-list">
         </div>
       </span>
    </div>
  </span
</main>

I am doing an scrapy project and what i want is to get all the first layer div.infocard-list and from those divs get its first layers div.infocard-list and so on.

Something like this:

def parse(content):
   depth_divs = []
   divs = content.xpath("get_layer_divs")
   if divs:
     for div in divs:
       depth_divs.append(div.id)
       next_layer_depth_list = parse(div)
       if next_layer_depth_list:
          depth_divs.append(next_layer_depth_list)
     
     return depth_divs

The above function should return this: ["1",["3","4"],"2",["5","6"]]

I tried to use css selector content.css(" > div.infocard-list"), but i get a syntax error because i don't provide any tag before ">" and i can't provide it because of the specific html that i am working on

scrapy beautifulsoup xpath css python
1 Answers

Try:

from bs4 import BeautifulSoup

html_doc = """
<main>
   <span>
     <div id="1" class="infocard-list">
       <span>
         <div id="3" class="infocard-list">
         </div>
       </span>
       <span>
         <div id="4" class="infocard-list">
         </div>
       </span>
    </div>
    <div id="2" class="infocard-list">
       <span>
         <div id="5" class="infocard-list">
         </div>
       </span>
       <span>
         <div id="6" class="infocard-list">
         </div>
       </span>
    </div>
  </span
</main>
"""

soup = BeautifulSoup(html_doc, "html.parser")


def get_tree(soup, seen):
    out = []
    for d in soup.find_all("div", class_="infocard-list"):
        if d not in seen:
            seen.add(d)
            out.append(d["id"])
            t = get_tree(d, seen)
            if t:
                out.append(t)
    return out


print(get_tree(soup, set()))

Prints:

['1', ['3', '4'], '2', ['5', '6']]
user_100359850
Related