How to configure partitionExprs in apache spark in JAVA

Viewed 21

I am using Apache Spark to read data from SQL Server to CSV with the below version details:

  • implementation 'com.microsoft.azure:spark-mssql-connector_2.12:1.2.0'
  • implementation 'org.apache.spark:spark-core_2.12:3.1.3'
  • implementation group: 'org.apache.spark', name: 'spark-sql_2.12', version: '3.1.3'

Looking for some way to load data from SQL Server in parallel to multiple CSV files having alphanumeric primary key. Although I am able to load with integer primary key with below code segments as an example:

     Dataset<Row> dataset = sqlContext.read().format("com.microsoft.sqlserver.jdbc.spark")
                .option("url", connectionUrl)
                .option("dbtable", "dbo.<table-name>")
                .option("user", "<user-name>")
                .option("password", "<password>")
                .option("fetchsize", 10000)
                .option("lowerBound", 1) //Min of ID column
                .option("upperBound", 15707106) //Max of ID column  
                .option("numPartitions", 5)
                .option("partitionColumn", "ID")
                .load();

I found some way from some blogs, it can be possible by defining:

  • partitionExprs
  • HashPartitioner

still didn't find a working JAVA example of how to configure all of these. Looking for support on the below queries:

  • How to configure the lower/upper bound in the column as of type VARCHAR or any expression? may not be required in that case but not sure.
  • Any JAVA sample of code will be helpful.
0 Answers
Related