Scraping a 'href' from a <span> with no class inside a<div>

user_14845591 0
Viewed 189

i am pretty new to scraping with BeatifulSoup and python and I am having some difficulties trying to get a href inside a span but it has no class.. The following portion of code is from a phpbb forum, i have no problem scraping all the href but for some reason i cannot figure it out how to grab what is inside the span..

<div class="col-md-48 post-text" data-topic="6693rw38" data-forum="2">
<br>
<br>
<a href="http://imgshare.net/img-5ba3dt3ad8a24.html" target="_blank" class="postlink" rel="nofollow"></a>
<br>
<br>
<a href="http://imgshare.net/img-5baefr1a51a49.html" target="_blank" class="postlink" rel="nofollow"></a>
<br>
<br>
<span>
    <a href="https://k2s.cc/file/5c745ce5g9193/toyota.mp4" target="_blank">https://k2s.cc/file/5c745ce5g9193/toyota.mp4</a>
</span>
<br>
<br>
<a href="http://imgshare.net/img-5ba34d1q805b8.html" target="_blank" class="postlink" rel="nofollow"></a>
<br>
<br>
<span>
    <a href="https://k2s.cc/file/b28gr283ef76e/ford.mp4" target="_blank">https://k2s.cc/file/b28gr283ef76e/ford.mp4</a>
</span>

This will give me all the 'href' inside the a tag:

url ='somephpbbforum.com'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'lxml')  

link = soup.find_all('div', class_ = 'col-md-48')

for div in link:          
    all_links = [link1['href'] for link1 in div.find_all('a')]
    print(all_links)

Thanks guys!

web-scraping python
1 Answers

You are probably looking for something like this (using css selctors):

all_links = [s['href'] for s in soup.select('div.col-md-48 > a[href]')]
all_links

output:

['http://imgshare.net/img-5ba3dt3ad8a24.html',
 'http://imgshare.net/img-5baefr1a51a49.html',
 'http://imgshare.net/img-5ba34d1q805b8.html']

Edit:

To get the text content of these nodes, use

all_links2 = [s.text for s in soup.select('div.col-md-48 > span > a[href]')]
all_links2

Output:

['https://k2s.cc/file/5c745ce5g9193/toyota.mp4',
 'https://k2s.cc/file/b28gr283ef76e/ford.mp4']
user_94480900
Related