PEM Certificate in Data Factory

Viewed 203

Can't find anything on this.

I am hitting an API for On the market, which is a UK real-estate website. As part of the auth, it requires us to submit a certificate & key. Docs here:

https://media.rightmove.co.uk/ps/pdf/guides/adf/Rightmove_Real_Time_Datafeed_Specification.pdf

(NB: Docs refer to Rightmove but On-the-market uses the same API).

I cannot see any functionality within Data Factory to do this, has anyone been able to overcome a similar challenge?

I have downloaded a PEM certificate from On-the-market. I've tried a Databricks script to test but I get an error message trying to load the certificate:

Python script:

import http.client
import json
import ssl
 
# Defining certificate related stuff and host of endpoint
certificate_file = "/mnt/data/certificates/otm/certificate17131114507144750725.pem"
certificate_secret= "<snipped>"
host = 'https://adfapi.rightmove.co.uk/v1'
 
# Defining parts of the HTTP request
request_url='/property/getbrandemail'
request_headers = {
    'Content-Type': 'application/json'
}
     
# Define the client certificate settings for https connection
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.load_cert_chain(certfile=certificate_file, password=certificate_secret)
 
# Create a connection to submit HTTP requests
connection = http.client.HTTPSConnection(host, port=443, context=context)
 
# Use connection to submit a HTTP POST request
connection.request(method="POST", url=request_url, headers=request_headers, body=json.dumps(request_body_dict))
 
# Print the HTTP response from the IOT service endpoint
response = connection.getresponse()
print(response.status, response.reason)
data = response.read()
print(data)

Error message:

FileNotFoundError: [Errno 2] No such file or directory --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) in 20 # Define the client certificate settings for https connection 21 context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) ---> 22 context.load_cert_chain(certfile=certificate_file, password=certificate_secret) 23 24 # Create a connection to submit HTTP requests

To be clear, the error relates to this line:

context.load_cert_chain(certfile=certificate_file, password=certificate_secret)

/mnt/data is a mount point I created and works fine in other notebooks. My idea was to get responses through a notebook and orchestrate that through ADF. Feels like a simpler solution should be available.

1 Answers

The problem is that you're using /mnt/data from the code that has no knowledge about DBFS (default filesystem). So you need to use local filesystem API - just add /dbfs to all paths, like, /dbfs/mnt/data/...

Related