How to get post id with Beautiful Soup web scraping?

user_15673147 0
Viewed 202

I'm creating a template for extracting news from websites for semantic analysis. I've just started learning to do web scraping with bs4 and selenium.

I could achieve to separate and get the title, subtitle and date from each news as following (already tested and working for any news):

title = soup.find_all('h1')[1].text
subtitle = soup.find_all('h2')[0].text
date = soup.find_all("span")[39].text

Now I'm trying to get the id from the post so I could save each news as a .txt file and the filename would be the id from the post.

If I use soup.find("article") I get the following:

<article class="article post" id="post-518551">
<div class="post-header">
<div class="category">
<a href="/entretenimento">Entretenimento</a> </div>
<h1 class="title">No Dia dos Namorados, Manu Gavassi assume relacionamento: “Eu te amo”</h1>
<h2 class="description">A cantora utilizou as redes sociais para publicar um clique fofo ao lado de Jullio</h2>
<span class="author">

(...)

I need only to extract the post id from the first line. I've tried several commands within soup.find but I couldn't debug how to get it. Everything I try returns to me the full article code.

Someone can help me?

Many thanks for the community!

web-scraping beautifulsoup python
1 Answers

If html_doc is your HTML snippet from the question, then:

soup = BeautifulSoup(html_doc, "html.parser")

article = soup.find("article")
post_id = article["id"].split("-")[1]
print(post_id)

Prints:

518551

If you want full ID (with post-):

post_id = article["id"]
user_100359850
Related