I am using BigQuery Spark Connector to write data to BigQuery using the Direct method. But I am getting the following exception that indicates that I am hitting the max Request Size limit of Storage Write API.
Caused by: java.util.concurrent.ExecutionException: com.google.cloud.spark.bigquery.repackaged.io.grpc.StatusRuntimeException: INVALID_ARGUMENT: MessageSize is too large. Max allow: 10000000 Actual: 10179670
at com.google.cloud.spark.bigquery.repackaged.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
at com.google.cloud.spark.bigquery.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
at com.google.cloud.spark.bigquery.repackaged.com.google.api.core.AbstractApiFuture.get(AbstractApiFuture.java:55)
at com.google.cloud.bigquery.connector.common.BigQueryDirectDataWriterHelper.validateAppendRowsResponse(BigQueryDirectDataWriterHelper.java:210)
... 21 more
Caused by: com.google.cloud.spark.bigquery.repackaged.io.grpc.StatusRuntimeException: INVALID_ARGUMENT: MessageSize is too large. Max allow: 10000000 Actual: 10179670
at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.storage.v1.StreamWriter.appendInternal(StreamWriter.java:301)
at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.storage.v1.StreamWriter.append(StreamWriter.java:292)
at com.google.cloud.bigquery.connector.common.BigQueryDirectDataWriterHelper.sendAppendRowsRequest(BigQueryDirectDataWriterHelper.java:187)
... 20 more
Is there any way to increase this limit?
If not how can ensure that that limit is not reached ? Even if I set spark.sql.files.maxPartitionBytes to 10MB, some partition still has data more than 10MB. Is there any other way to achieve this ?
It looks like there is no way to update the request size as mentioned here: https://www.googlecloudcommunity.com/gc/Data-Analytics/How-to-increase-limit-of-Storage-Write-API/m-p/446668#M517