Google BigQuery fails with "Resources exceeded during query execution: UDF out of memory" when loading Parquet file

user_12973929 0
Viewed 2494

We use the BigQuery Java API to upload data from local data source as described here. When uploading a Parquet file with 18 columns (16 string, 1 float64, 1 timestamp) and 13 Mio rows (e.g. 17GB of data) the upload fails with the following exception:

Resources exceeded during query execution: UDF out of memory.; Failed to read Parquet file . This might happen if the file contains a row that is too large, or if the total size of the pages loaded for the queried columns is too large.

However when uploading the same data using CSV (17.5GB of data) the upload succeeds. My questions are:

  • What is the difference when uploading Parquet or CSV?
  • What query is executed during upload?
  • Is it possible to increase the memory for this query?

Thanks Tobias

google-bigquery
2 Answers

Parquet is columnar data format, which means that loading data requires reading all columns. In parquet, columns are divided into pages. BigQuery keeps entire uncompressed pages for each column in memory while reading data from them. If the input file contains too many columns, BigQuery workers can hit Out of Memory errors.

Even when a precise limit is not enforced as it happens with other formats, it is recommended that records should in the range of 50 Mb, loading larger records may lead to resourcesExceeded errors.

Taking into account the above considerations, it would be great to clarify the following points:

  1. What is the maximum size of rows in your Parquet file?
  2. What is the maximum page size per column?

This info can be retrieved by publicly available tool.

If you think about increasing the alocated memory for queries, you need to read about Bigquery slots.

user_70900160

In my case, I ran bq load --autodetect --source_format=PARQUET ... which failed with the same error (resources exceeded during query execution). Finally, I had to split the data into multiple Parquet files so that they would be loaded in batches.

user_12270570
Related