Unable to select all tag elements at once using css selectors in bs4

Admin 1
Viewed 45

I'm getting only first tags value as text from each quote list using select_one but when I use select() in css selectors then pandas DataFrame throwing error: 'arrays must all be same length'.

For example : Tags: change deep-thoughts thinking world

I expect :change deep-thoughts thinking world but I'm getting only change

CODE:

import requests 
import bs4
import pandas as pd

res = requests.get("http://quotes.toscrape.com/") 

soup = bs4.BeautifulSoup(res.text,'lxml') 

all_quote = []
all_author = []
all_tag = []

for element in soup.select('.quote'):
    quote = element.select_one('span.text').text
    all_quote.append(quote)
    
    author = element.select_one('small.author').text
    all_author.append(author)
    
    tag = element.select_one('a.tag').text
    all_tag.append(tag)
  
df = pd.DataFrame({'all_quote': all_quote,'all_author':all_author, 'all_tag': all_tag}) 
print(df)

OUTPUT:

 all_quote  ...                          all_tag
0  “The world as we have created it is a process ...  ...                           change
1  “It is our choices, Harry, that show what we t...  ...                        abilities
2  “There are only two ways to live your life. On...  ...                    inspirational
3  “The person, be it gentleman or lady, who has ...  ...                        aliteracy
4  “Imperfection is beauty, madness is genius and...  ...                      be-yourself
5  “Try not to become a man of success. Rather be...  ...                        adulthood
6  “It is better to be hated for what you are tha...  ...                             life
7  “I have not failed. I've just found 10,000 way...  ...                           edison
8  “A woman is like a tea bag; you never know how...  ...  misattributed-eleanor-roosevelt
9  “A day without sunshine is like, you know, nig...  ...                            humor

[10 rows x 3 columns]
pandas web-scraping beautifulsoup css-selectors python
1 Answers

You can use list comprehension when selecting all tags from the quote. For example:

import requests
import bs4
import pandas as pd

res = requests.get("http://quotes.toscrape.com/")

soup = bs4.BeautifulSoup(res.text, "lxml")

all_quote = []
all_author = []
all_tag = []

for element in soup.select(".quote"):
    quote = element.select_one("span.text").text
    all_quote.append(quote)

    author = element.select_one("small.author").text
    all_author.append(author)

    tag = [e.text for e in element.select("a.tag")]  # <-- select all tags
    all_tag.append(tag)

df = pd.DataFrame(
    {"all_quote": all_quote, "all_author": all_author, "all_tag": all_tag}
)
print(df)

Prints:

                                                                                                                             all_quote         all_author                                         all_tag
0                  “The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”    Albert Einstein        [change, deep-thoughts, thinking, world]
1                                                “It is our choices, Harry, that show what we truly are, far more than our abilities.”       J.K. Rowling                            [abilities, choices]
2  “There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”    Albert Einstein  [inspirational, life, live, miracle, miracles]
3                             “The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”        Jane Austen              [aliteracy, books, classic, humor]
4                      “Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”     Marilyn Monroe                    [be-yourself, inspirational]
5                                                                  “Try not to become a man of success. Rather become a man of value.”    Albert Einstein                     [adulthood, success, value]
6                                                   “It is better to be hated for what you are than to be loved for what you are not.”         André Gide                                    [life, love]
7                                                                    “I have not failed. I've just found 10,000 ways that won't work.”   Thomas A. Edison   [edison, failure, inspirational, paraphrased]
8                                                “A woman is like a tea bag; you never know how strong it is until it's in hot water.”  Eleanor Roosevelt               [misattributed-eleanor-roosevelt]
9                                                                                   “A day without sunshine is like, you know, night.”       Steve Martin                        [humor, obvious, simile]
user_100359850
Related