I am using Apache Spark to read data from SQL Server to CSV with the below version details:
- implementation 'com.microsoft.azure:spark-mssql-connector_2.12:1.2.0'
- implementation 'org.apache.spark:spark-core_2.12:3.1.3'
- implementation group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.1.3'
Looking for some way to load data from SQL Server in parallel to multiple CSV files having alphanumeric primary key. Although I am able to load with integer primary key with below code segments as an example:
Dataset<Row> dataset = sqlContext.read().format("com.microsoft.sqlserver.jdbc.spark")
.option("url", connectionUrl)
.option("dbtable", "dbo.<table-name>")
.option("user", "<user-name>")
.option("password", "<password>")
.option("fetchsize", 10000)
.option("lowerBound", 1) //Min of ID column
.option("upperBound", 15707106) //Max of ID column
.option("numPartitions", 5)
.option("partitionColumn", "ID")
.load();
I found some way from some blogs, it can be possible by defining:
- partitionExprs
- HashPartitioner
still didn't find a working JAVA example of how to configure all of these. Looking for support on the below queries:
- How to configure the lower/upper bound in the column as of type VARCHAR or any expression? may not be required in that case but not sure.
- Any JAVA sample of code will be helpful.