Web scrape up to a specific point in the html then stop

user_11400264 0
Viewed 196

I am scraping li tags and want to stop at a specific place in the html. I do not want the li tags that are after this point in the html.

I am usnig python 3.7, beautifulsoup4, here is a link to example site data

<li class="s-item" data-view="mi:1686|iid:1">...</li>
<li class="s-item" data-view="mi:1686|iid:2">...</li>
<li class="s-item" data-view="mi:1686|iid:3">...</li>
<div class="srp-river-answer srp-river-answer--REWRITE_START" data-view="mi:1706|iid:1">...</div>
<li class="s-item" data-view="mi:1686|iid:4">...</li>
<li class="s-item" data-view="mi:1686|iid:5">...</li>
<li class="s-item" data-view="mi:1686|iid:6">...</li>

Goal is to get all li tags before the div statement and none after the div statement. The number of li tags before and after vary by page. I have tried and cannot find a way to have beautifulsoup scrape until a point in the html.

I currently retrieve all li tags with the following

only_li = SoupStrainer('li')
soup_li = BeautifulSoup(response.text, 'lxml', parse_only = only_li)
sale_details = soup_li.find_all('li', attrs={'class':'s-item','data-view':True})
for sale in sale_details:
    some scrape code
web-scraping beautifulsoup html python
1 Answers

Based on the html you posted in the question, you can use CSS selector li.s-item:not(div.srp-river-answer ~ li.s-item):

from bs4 import BeautifulSoup

txt = '''
<li class="s-item" data-view="mi:1686|iid:1">tag 1</li>
<li class="s-item" data-view="mi:1686|iid:2">tag 2</li>
<li class="s-item" data-view="mi:1686|iid:3">tag 3</li>
<div class="srp-river-answer srp-river-answer--REWRITE_START" data-view="mi:1706|iid:1">...</div>
<li class="s-item" data-view="mi:1686|iid:4">tag 4</li>
<li class="s-item" data-view="mi:1686|iid:5">tag 5</li>
<li class="s-item" data-view="mi:1686|iid:6">tag 6</li>
'''

soup = BeautifulSoup(txt, 'html.parser')


for li in soup.select('li.s-item:not(div.srp-river-answer ~ li.s-item)'):
    print(li.text)

Prints:

tag 1
tag 2
tag 3

EDIT: To get data from the link you posted:

import requests
from bs4 import BeautifulSoup

url = 'https://www.ebay.com/sch/i.html?_odkw=W10083957V&_sop=12&_sadis=15&_dmd=1&LH_Complete=1&LH_ItemCondition=4&LH_BIN=1&LH_Sold=1&_osacat=0&_ipg=200&_stpos=23114-3265&_from=R40&_trksid=m570.l1313&_nkw=W10083957V&_sacat=0'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for li in soup.select('li[data-view].s-item:not(div.srp-river-answer--REWRITE_START ~ li.s-item)'):
    print(li.h3.get_text(strip=True))

Prints:

Maytag, Kenmore, Amana & Jenn Air Chopper Blade Assembly W10083957V - W10083957
user_100359850
Related