Finding the row count of data of a non-query job in Google BigQuery

Viewed 201

I have this code to load CSV data from Cloud Storage into a new BigQuery table. The load operation completes successfully but I need metrics about data count.

import com.google.cloud.bigquery.*;
import com.google.cloud.bigquery.JobInfo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class LoadCsvFromGcs {

    private static final Logger LOGGER = LoggerFactory.getLogger(LoadCsvFromGcs.class);
    private static long csvRowCount;

    public static void loadCsvFromGcs(
            String datasetName, String tableName, String sourceUri, Schema schema) {
        try {
            BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
            CsvOptions csvOptions = CsvOptions.newBuilder().setSkipLeadingRows(1).build();

            TableId tableId = TableId.of(datasetName, tableName);
            LoadJobConfiguration loadConfig =
                    LoadJobConfiguration.newBuilder(tableId, sourceUri, csvOptions).setSchema(schema)
                            .setMaxBadRecords(999999999).build();
            Job job = bigquery.create(JobInfo.of(loadConfig));
            job = job.waitFor();

            csvRowCount = job.getQueryResults().getTotalRows();
            LOGGER.info(Long.toString(csvRowCount));

            if (job.isDone()) {
                LOGGER.info(sourceUri + " CSV from GCS successfully added during load append job");
            } else {
                LOGGER.error("BigQuery was unable to load into the table due to an error:"
                        + job.getStatus().getError() + " , " + sourceUri);
                System.exit(-1);
            }
        } catch (BigQueryException | InterruptedException e) {
            LOGGER.error(sourceUri + " Column not added during load append \n" + e);
            System.exit(-1);
        }
    }
}

I took a look at the Google BigQuery documentation and saw that the metrics of the job class can be retrieved with the get functions. However, when I try, I get the following error.

Exception in thread "main" java.lang.UnsupportedOperationException: Getting query results is supported only for QUERY jobs

How do I find out metrics like how many rows of data are loaded for non-query jobs?

0 Answers
Related