Python/AWS Neptune getting "tornado.httpclient.HTTPError: HTTP 403: Forbidden"

user_1187968 0
Viewed 451

So I'm using a bastion host/SSH tunnel to connect from my local computer to AWS Neptune.

ssh -N -i /Users/user1/.ssh/id_rsa -L 8182:my.xxx.us-east-1.neptune.amazonaws.com:8182 user1@transporter-int.mycloud.com

I did a simple Neptune connection test with gremlin.

from gremlin_python.process.graph_traversal import __
from gremlin_python.structure.graph import Graph
from gremlin_python.process.strategies import *
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.traversal import T

graph = Graph()

wss = 'wss://{}:{}/gremlin'.format('localhost', 8182)
remoteConn = DriverRemoteConnection(wss, 'g')
g = graph.traversal().withRemote(remoteConn)

print(g.V().limit(2).toList())
remoteConn.close()

And getting this error:

*aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host 
localhost:8182 ssl:True [SSLCertVerificationError: (1, "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'localhost'. (_ssl.c:1124)")]*

With @Taylor Riggan's suggestion, I update the /etc/hosts on my mac to the following:

Switched to use Python version 3.6.12, and gremlin-python version 3.4.10

127.0.0.1 localhost my.cluster-xxx.us-east-1.neptune.amazonaws.com

ran the following command to flush the hosts setting

sudo dscacheutil -flushcache

updated this line in the source code

wss = 'wss://{}:{}/gremlin'.format('my.cluster-xxx.us-east-1.neptune.amazonaws.com', 8182).

and now getting the following error, and the tornado version 4.5.3

  File "/Users/user1/myproj/tests/graph/venv/lib/python3.6/site-packages/gremlin_python/driver/client.py", line 148, in submitAsync
    return conn.write(message)
  File "/Users/user1/myproj/tests/graph/venv/lib/python3.6/site-packages/gremlin_python/driver/connection.py", line 55, in write
    self.connect()
  File "/Users/user1/myproj/tests/graph/venv/lib/python3.6/site-packages/gremlin_python/driver/connection.py", line 45, in connect
    self._transport.connect(self._url, self._headers)
  File "/Users/user1/myproj/tests/graph/venv/lib/python3.6/site-packages/gremlin_python/driver/tornado/transport.py", line 41, in connect
    lambda: websocket.websocket_connect(url, compression_options=self._compression_options))
  File "/Users/user1/myproj/tests/graph/venv/lib/python3.6/site-packages/tornado/ioloop.py", line 576, in run_sync
    return future_cell[0].result()
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
amazon-neptune gremlin amazon-web-services ssl python
2 Answers

Easiest workaround for this is to add an entry in your /etc/hosts file on your dev desktop to resolve the Neptune endpoint to localhost. Then the cert validation should go through.

Ex:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost myneptune-cluster.region.neptune.amazonaws.com
255.255.255.255 broadcasthost
::1             localhost
user_101303720

Check the guide Connect to AWS Neptune from the local system

Connect to AWS Neptune from the local system

There are many ways to connect to Amazon Neptune from outside of the VPC, such as setting up a load balancer or VPC peering.

Amazon Neptune DB clusters can only be created in an Amazon Virtual Private Cloud (VPC). One way to connect to Amazon Neptune from outside of the VPC is to set up an Amazon EC2 instance as a proxy server within the same VPC. With this approach, you will also want to set up an SSH tunnel to securely forward traffic to the VPC.

Part 1: Set up a EC2 proxy server.

Launch an Amazon EC2 instance located in the same region as your Neptune cluster. In terms of configuration, Ubuntu can be used. Since this is a proxy server, you can choose the lowest resource settings.

Make sure the EC2 instance is in the same VPC group as your Neptune cluster. To find the VPC group for your Neptune cluster, check the console under Neptune > Subnet groups. The instance's security group needs to be able to send and receive on port 22 for SSH and port 8182 for Neptune. See below for an example security group setup.

Image

Lastly, make sure you save the key-pair file (.pem) and note the directory for use in the next step.

Part 2: Set up an SSH tunnel.

This step can vary depending on if you are running Windows or MacOS.

  1. Modify your hosts file to map localhost to your Neptune endpoint.

    Windows: Open the hosts file as an Administrator (C:\Windows\System32\drivers\etc\hosts)

    MacOS: Open Terminal and type in the command: sudo nano /etc/hosts

    Add the following line to the hosts file, replacing the text with your Neptune endpoint address.

    127.0.0.1 localhost YourNeptuneEndpoint

    Open Command Prompt as an Administrator for Windows or Terminal for MacOS and run the following command. For Windows, you may need to run SSH from C:\Users\YourUsername\

    ssh -i path/to/keypairfilename.pem ec2-user@yourec2instanceendpoint -N -L 8182:YourNeptuneEndpoint:8182

    The -N flag is set to prevent an interactive bash session with EC2 and to forward ports only. An initial successful connection will ask you if you want to continue connecting? Type yes and enter.

    To test the success of your local graph-notebook connection to Amazon Neptune, open a browser and navigate to:

    https://YourNeptuneEndpoint:8182/status

    You should see a report, similar to the one below, indicating the status and details of your specific cluster:

     {
   "status": "healthy",
   "startTime": "Wed Nov 04 23:24:44 UTC 2020",
   "dbEngineVersion": "1.0.3.0.R1",
   "role": "writer",
   "gremlin": {
     "version": "tinkerpop-3.4.3"
   },
   "sparql": {
     "version": "sparql-1.1"
   },
   "labMode": {
     "ObjectIndex": "disabled",
     "DFEQueryEngine": "disabled",
     "ReadWriteConflictDetection": "enabled"
   }
 }

Close Connection

When you're ready to close the connection, use Ctrl+D to exit.
user_33914900
Related