Getting urllib2.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:661)> in mechanize open method

user_6257144 0
Viewed 4558

I am getting urllib2.URLError: error while calling mechanize.browser.open('my https site').

I searched the web but nothing worked for me.

Here is my code:

import ssl
try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    # Legacy Python that doesn't verify HTTPS certificates by default
    pass
else:
    # Handle target environment that doesn't support HTTPS verification
    ssl._create_default_https_context = _create_unverified_https_context
import mechanize
import operator
from bs4 import BeautifulSoup
import os
myBrowser = mechanize.Browser()
myBrowser.set_handle_robots(False)
myBrowser.set_handle_refresh(False)
myBrowser.open("https://uwp.puchd.ac.in/common/viewmarks.aspx")

Here is the error I am getting:

Traceback (most recent call last):
  File "C:/Users/Himanshu/Desktop/UIET Rank system.py", line 27, in <module>
    myBrowser.open("https://uwp.puchd.ac.in/common/viewmarks.aspx")
  File "C:\Python27\lib\site-packages\mechanize\_mechanize.py", line 254, in open
    return self._mech_open(url_or_request, data, timeout=timeout)
  File "C:\Python27\lib\site-packages\mechanize\_mechanize.py", line 284, in _mech_open
    response = UserAgentBase.open(self, request, data)
  File "C:\Python27\lib\site-packages\mechanize\_opener.py", line 195, in open
    response = urlopen(self, req, data)
  File "C:\Python27\lib\site-packages\mechanize\_urllib2_fork.py", line 352, in _open
    '_open', req)
  File "C:\Python27\lib\site-packages\mechanize\_urllib2_fork.py", line 340, in _call_chain
    result = func(*args)
  File "C:\Python27\lib\site-packages\mechanize\_urllib2_fork.py", line 1215, in https_open
    return self.do_open(conn_factory, req)
  File "C:\Python27\lib\site-packages\mechanize\_urllib2_fork.py", line 1160, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:661)>

Process finished with exit code 1

Other information:

import ssl
print ssl.OPENSSL_VERSION

output>> OpenSSL 1.0.2j  26 Sep 2016

python version

Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32

Is there any way to bypass this error?

Note:

  • I want to use mechanize only, since my application is ready and it used to work a year before but now it's not working and I don't want to change the entire code again.
  • I am using Pycharm on windows.
  • Please try to open this webpage which I am trying to open it shows "Insecure connection" in chrome also and we need to proceed to the webpage. It might be the problem. Also, I don't have the certificate for this webpage.
  • My application has nothing to do with security, so it will be perfectly fine to set SSL verification to false (I tried to do so from other posts, but it didn't work for me) or something like that. Only the goal is that application should work.
mechanize urllib2 ssl python
2 Answers

The problem with this site is not the certificate validation since you have successfully switched it off. The problem instead is that the site only supports ciphers which are no longer considered secure, i.e. 3DES and RC4 based ciphers. The default ciphers in the ssl library don't include these ciphers for security reasons.

To add support for these ciphers you can manually set the default cipher set. The following line sets DES-CBC3-SHA as the offered cipher. This way you can access the broken site:

ssl._DEFAULT_CIPHERS = ('DES-CBC3-SHA')
myBrowser = mechanize.Browser()
...

Note that you should use this setting only for the specific site. While it might in theory be also possible to just set a larger cipher set for _DEFAULT_CIPHERS to handle all sites, this specific site suffers from additional problems: it looks like it will fail with the TLS handshake even if DES-CBC3-SHA is included in the offered cipher set but if newer ciphers (like GCM) are offered before DES-CBC3-SHA.

user_30810180

This error doc at Fix urlopen error EOF occurred in violation of protocol (_ssl.c:719) Error – Python Web Crawler Tutorial says that the error can be thrown when the SSL version is outdated. Try updating SSL to what your python version allows. This is also the answer on Stack Overflow at Python and SSL -- EOF occurred in violation of protocol.

Since in a legacy project that usually might throw such an error, updating SSL would also mean updating Python, you might try running the code again instead. In my use case of URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:661), the error was not thrown anymore during the second run.

user_111548410
Related