Struggling with BeautifulSoup and tags

user_11515076 0
Viewed 70

I hate to trouble anyone with this, but I've been on this issue for days.

Basically, I want to scrape the Psychological Torture Methods from this web page: https://en.m.wikipedia.org/wiki/List_of_methods_of_torture

This is the exact information I would like to acquire:

Ego-Fragmentation

Learned Helplessness

Chinese water torture

Welcome parade (torture)

And below is my code:

from bs4 import BeautifulSoup
import requests

URL = 'https://en.m.wikipedia.org/wiki/List_of_methods_of_torture'
page = requests.get(URL)

html_soup = BeautifulSoup(page.content, 'html.parser')
type(html_soup)


print (html_soup.find("div", class_="mw-parser-output").find_all(text=True, recursive=False) )

I'm sure there is an easy fix to this that I can't see. One you look at the sites html, you'll probably find the answer.

Best wishes, truly.

Have a Beautiful day!

HomeMadeMusic.

beautifulsoup python
3 Answers

When in doubt, brute force it and pretend like you'll come back to it later

from bs4 import BeautifulSoup
import requests

URL = 'https://en.m.wikipedia.org/wiki/List_of_methods_of_torture'
page = requests.get(URL)

html_soup = BeautifulSoup(page.content, 'html.parser')

sections = html_soup.find_all("section")
torture_methods = sections[1].find_all("li")
torture_method_names = list(map(lambda x: x.text, torture_methods))
print(torture_method_names)

Prints:

['Ego-Fragmentation', 'Learned Helplessness', 'Chinese water torture', 'Welcome parade (torture)']
user_126961400

Try this . Your expected output is in under section

from bs4 import BeautifulSoup
import requests

URL = 'https://en.m.wikipedia.org/wiki/List_of_methods_of_torture'
page = requests.get(URL)

html_soup = BeautifulSoup(page.content, 'html.parser')
print(html_soup.prettify())


print ([x.text for x in html_soup.find("section", class_="mf-section-1").find_all('a')])
user_48351220

There were a couple of issues here: First the recursive=False parameter means that you will only get the text that is directly inside the node you selected. You won't get the text from its subnodes. As there is no text directly inside this div element, the method returns an empty list.

Second problem: the div you selected doesn't only contain the "Psychological Torture Methods" section but also the other sections of the page as well as the disclaimer displayed at the beginning of the article. To get the information you need, you should only get the content of the section node whose class is mf-section-1

Solution

I just tweaked your code to print the information you needed. I had to use the lstrip method to remove an unnecessary line break.

from bs4 import BeautifulSoup
import requests

URL = 'https://en.m.wikipedia.org/wiki/List_of_methods_of_torture'
page = requests.get(URL)

html_soup = BeautifulSoup(page.content, 'html.parser')
type(html_soup)

print (''.join(html_soup.find("section", class_="mf-section-1").findAll(text=True)).lstrip("\n"))

Output

Ego-Fragmentation
Learned Helplessness
Chinese water torture
Welcome parade (torture)
user_98524380
Related