Google Dataproc to SQL Server(based on centos 7) connection error?

user_15233028 0
Viewed 622

I got stuck into an issue which already has wasted 3 days of mine. I have a dataproc cluster 1.5 and i also did setup SQL Server on google VM running centos 7 OS. But i am unable to connect SQL Server through pyspark from dataproc cluster. You can find the error snapshot in the attachment. SSL encryption is disabled on SQL server. I can access SQL server through sqlcmd(installed on dataproc cluster) and also through PYMSSQL library from dataproc cluster. But not with pyspark. The same error occurs while trying to access MSSQL from Sqoop as well. Kindly, guide me i have tried all possible solution available on internet but still no luck for me. Thanks in advance. My Connection String is:

df = spark.read.format("jdbc") \
.option("url", "jdbc:sqlserver://x.x.x.x:1433;encrypt=false;databaseName=gcp") \
.option("dbtable", "xxx") \
.option("user", "xxx") \
.option("password", "xxxx") \
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()

Connection Error Snapshot

sqoop google-cloud-dataproc pyspark database-connection sql-server
1 Answers

This could happen because Dataproc uses Conscrypt by default to improve performance.

Depending on MS SQL JDBC deriver version that you use it can have bugs that lead to failures when Conscrypt is used.

To workaround this issue try to disable Conscrypt during Dataproc cluster creation via cluster properties:

gcloud dataproc clusters create $CLUSTER_NAME \
  --properties=dataproc:dataproc.conscrypt.provider.enable=false
user_32276930
Related