Why do I get an empty list while scraping website and end in type error?

Fırat Kaya 0
Viewed 59

Hi i am trying to do web scraping in pyhton but it returns empty string. I couldn't solve the problem as I am a beginner. As far as I researched, I get a

TypeError: list indices must be integers or slices, not str

when I do it, can you help?

When I use this shape it returns empty data

import requests
from bs4 import  BeautifulSoup
import pandas as pd
from urllib import response


kitapurl = "https://1000kitap.com/alintilar"

response = requests.get(kitapurl)
soup = BeautifulSoup(response.content,"html.parser")
gelen_ana_veri = soup.find_all('span',attrs={'class':'text-alt'})
print(gelen_ana_veri)

but if i do it this way i get type error

import requests
from bs4 import  BeautifulSoup
import pandas as pd
from urllib import response


kitapurl = "https://1000kitap.com/alintilar"

response = requests.get(kitapurl)
soup = BeautifulSoup(response.content,"html.parser")
gelen_ana_veri= soup.find_all('meta',attrs={'property':'og:description'})['content']
print(gelen_ana_veri)

The error i got

    Traceback (most recent call last):
  File "c:\Users\fratk\OneDrive\Belgeler\twbot\1001kitap.py", line 11, in <module>
    elem = soup.find_all('meta',attrs={'property':'og:description'})['content']
TypeError: list indices must be integers or slices, not str
typeerror beautifulsoup web-scraping python-3.x python
2 Answers

Always and first of all check the response to your request or / and your soup.

  • is the request successful?
  • are the expected elements included?
  • is there information that suggests that content is being withheld?
  • ...

The website is protected via cloudflare

Attention Required! | Cloudflare Please enable cookies. Sorry, you have been blocked You are unable to access1000kitap.comWhy have I been blocked? This website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution...

So this page does not want to be scraped and you should respect that - From a technical point of view there is still an option to access the page and its contents e.g. with use of cloudscraper.

Anyway, the ResultSet of your find_all() cannot be treated as a dict, a solution would be to iterate or as in the case simply use find() as there is only one descriptive meta - Also use get() to avoid error if attribute is not available:

soup.find('meta',attrs={'property':'og:description'}).get('content')

Example

import cloudscraper
from bs4 import  BeautifulSoup

kitapurl = "https://1000kitap.com/alintilar"
scraper = cloudscraper.create_scraper(browser={'browser': 'firefox','platform': 'windows','mobile': False})

response = scraper.get(url).content
soup = BeautifulSoup(response,"html.parser")
gelen_ana_veri= soup.find('meta',attrs={'property':'og:description'}).get('content')
print(gelen_ana_veri)
HedgeHog0

That is because content doesn't exist.

If you try elem = soup.find_all('meta',attrs={'property':'og:description'}) you will get nothing or an empty list []

You would also get the TypeError: list indices must be integers or slices, not str if you were to try gelen_ana_veri = soup.find_all('span',attrs={'class':'text-alt'})['content']

first check to see if the element you are looking for exists with soup.find_all() function in a more broad sense. Then if you find your element, then try to drill down more of its classes, or properties.

Here is a basic example of trying to find what you are looking for:

gelen_ana_veri = soup.find_all('meta')
print(gelen_ana_veri)

This will output a list of every meta tag and then you can see of any specific tags you are looking for.

PythonKiddieScripterX0
Related