just started web scraping in python and I'm facing some problems.
I started using Selenium to download a webpage's source and saving it:
from selenium import webdriver
driver= webdriver.Firefox()
driver.get("https://www.website.com")
f=open('output.txt','w')
f.write(driver.page_source.encode('utf-8'))
f.close()
driver.quit()
Everything worked fine, but Selenium takes too much time, so I first turned to mechanize, to obtain the page source:
import mechanize
browser = mechanize.Browser()
browser.set_handle_robots(False)
cookies = mechanize.CookieJar()
browser.set_cookiejar(cookies)
browser.addheaders = [('User-agent', 'Mozilla/5.0')]
browser.set_handle_refresh(False)
browser.open("https://www.website.com")
Here comes the problem: if I try finding a particular div by it's id, it returns me none:
from bs4 import BeautifulSoup as BS
soup= BS(browser.response().read(),'lxml')
print(soup.find(id="div_id"))
Although if I inspect the source code obtained with mechanize with a regular text editor I can find it. It's something like:
<div id="div_id" data referrer="div_id">
This div has many other child elements, it's situated about 1/5 "into" the code, and the complete source code is about 500kb.If I try instead looking for other div near, no luck either. Whereas if I look for some div near the beginning of the source code, it finds it.And what's more interesting, if I try looking for the same div (with BS), in the source code obtained with Selenium, instead of the one obtained with Mechanize, it's able to find it, although the div seems completely the same by inspection with a text editor.
I tried with all of BS supported parsers, with no luck. So I thought it maybe had something to do with BS, and I tried to do the same with lxml:
from lxml import etree
parser= etree.HTMLParser()
tree= etree.parse(open('source.txt'),parser)
results= tree.xpath('//div[@id="div_id"]')
print(etree.tostring(results[0]))
As with BS, it was able to find the div in the source code obtained with Selenium, but not with Mechanize. So I thought it could have something to do with Mechanize, and turned to use Requests:
import requests
from fake_useragent import UserAgent
ua=UserAgent()
url= 'https://www.website.com'
headers= {'User-agent': str(ua.chrome)}
page = requests.get(url, headers=headers)
When looked in page.content for the div, with either BS or lxml, again no luck.It happens whether I analyze directly the response or whether if I save it into a file and then analyze the file.
And I thinks that's about it... I also tried encoding the Mechanize and Requests responses, as I saw I had done it with Selenium, but no change. I also tried using other BS version (3.x), no change.
To summarize: - If I look for the div with BS or lxml into the source code obtained via Selenium, it finds it.With the other ones, no. - If I look for other divs at the beginning of the source code, BS and lxml find it, independent of the method used to obtained the code. - On inspection, the div is there in every case.
Versions used: -python: 2.7.9 -BeautifulSoup: 4.6.0 -Mechanize: 0.3.5 -Requests: 2.18.4 -Selenium: 3.5.0 -lxml: 4.0.0 -OS: linux debian
Thank you.