Scraping multiple wikitables using Python

Viewed 64

I am very beginner to Python. I have a task to scrape information table from wikipedia page. I would like to scrape using the below code:

from pandas.io.html import read_html
page = requests.get('https://de.wikipedia.org/wiki/Köln')
wikitables = read_html(page, attrs={"class":"hintergrundfarbe5 float-right toptextcells infobox"})
print("Extracted {num} wikitables".format(num=len(wikitables)))

wikitables[0]

But I get the below error due to the special character in the Url as Köln: Please help me where to do the modifications in the program to scrape the information.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-168-d9bd1e1d7548> in <module>
      2 page = requests.get('https://de.wikipedia.org/wiki/Köln')
      3 Soup = BeautifulSoup(page.content)
----> 4 wikitables = read_html(page, attrs={"class":"hintergrundfarbe5 float-right toptextcells infobox"})
      5 print("Extracted {num} wikitables".format(num=len(wikitables)))
      6 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\html.py in read_html(io, match, flavor, header, index_col, skiprows, attrs, parse_dates, tupleize_cols, thousands, encoding, decimal, converters, na_values, keep_default_na, displayed_only)
   1092                   decimal=decimal, converters=converters, na_values=na_values,
   1093                   keep_default_na=keep_default_na,
-> 1094                   displayed_only=displayed_only)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\io\html.py in _parse(flavor, io, match, attrs, encoding, displayed_only, **kwargs)
    914             break
    915     else:
--> 916         raise_with_traceback(retained)
    917 
    918     ret = []

~\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\compat\__init__.py in raise_with_traceback(exc, traceback)
    418         if traceback == Ellipsis:
    419             _, _, traceback = sys.exc_info()
--> 420         raise exc.with_traceback(traceback)
    421 else:
    422     # this version of raise is a syntax error in Python 3

TypeError: Cannot read object of type 'Response'
1 Answers

This has nothing to do with beautiful Köln...

You need to change

wikitables = read_html(page, attrs={"..."})

to

wikitables = read_html(page.text, attrs={"..."})

and it should work.

Related