I have this code to load CSV data from Cloud Storage into a new BigQuery table. The load operation completes successfully but I need metrics about data count.
import com.google.cloud.bigquery.*;
import com.google.cloud.bigquery.JobInfo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LoadCsvFromGcs {
private static final Logger LOGGER = LoggerFactory.getLogger(LoadCsvFromGcs.class);
private static long csvRowCount;
public static void loadCsvFromGcs(
String datasetName, String tableName, String sourceUri, Schema schema) {
try {
BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
CsvOptions csvOptions = CsvOptions.newBuilder().setSkipLeadingRows(1).build();
TableId tableId = TableId.of(datasetName, tableName);
LoadJobConfiguration loadConfig =
LoadJobConfiguration.newBuilder(tableId, sourceUri, csvOptions).setSchema(schema)
.setMaxBadRecords(999999999).build();
Job job = bigquery.create(JobInfo.of(loadConfig));
job = job.waitFor();
csvRowCount = job.getQueryResults().getTotalRows();
LOGGER.info(Long.toString(csvRowCount));
if (job.isDone()) {
LOGGER.info(sourceUri + " CSV from GCS successfully added during load append job");
} else {
LOGGER.error("BigQuery was unable to load into the table due to an error:"
+ job.getStatus().getError() + " , " + sourceUri);
System.exit(-1);
}
} catch (BigQueryException | InterruptedException e) {
LOGGER.error(sourceUri + " Column not added during load append \n" + e);
System.exit(-1);
}
}
}
I took a look at the Google BigQuery documentation and saw that the metrics of the job class can be retrieved with the get functions. However, when I try, I get the following error.
Exception in thread "main" java.lang.UnsupportedOperationException: Getting query results is supported only for QUERY jobs
How do I find out metrics like how many rows of data are loaded for non-query jobs?