Website doesn't allow right click and web scrape does not show text between body tags

user_11536447 0
Viewed 687

I am trying to web scrape a web page. However, the website itself does not allow a right click functionality to inspect the element or page source. In addition, when I print the parsed html text, the result only has body tags and does not show the information inside the body tags.

from selenium import webdriver
import bs4 as bs
import urllib.request

driver = webdriver.Chrome(executable_path=r'PATH')
driver.get("website.com")

assert "Title" in driver.title

sauce = urllib.request.urlopen('website.com').read()

soup = bs.BeautifulSoup(sauce, 'html.parser')
print(soup)

I expect to get the actual HTML (specifically HTML between the body tags) from the soup; however, I only get 

<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
<title>Title</title>

</head>
<body></body>
</html>

I am relatively new to webscraping; however, I never had this problem before. Is it specific to the site? When I use a chrome extension, I am able to allow for right clicking, and only then, can I inspect the page source (and there is HTML and tags within the body tags).

python-requests selenium urllib html python
2 Answers

You could try getting the body tag information using just Python / Selenium and see if the page source is any different than what you posted in your question. This will help narrow down the problem.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("website.com")

html = driver.page_source

print(html)

If this produces the same results as HTML you posted in your question, then the issue is probably with the website itself and not your code.

user_110197430

This might not answer the question but it should get your context menu back:

driver.execute_script("""
  window.addEventListener('contextmenu', e => {
    e.stopPropagation();
    e.stopImmediatePropagation();
  }, true)
""")
user_9660230
Related