BigQuery Loops VS Multiple Single Queries

Viewed 20

I'm using Python to:

  1. Fetch a list of values from column in table A
  2. Iterate over the list and query table B for each value

This seems like an awful pattern, since BigQuery isn't transactional. I know we can loop in BigQuery. But I'm not sure if there is any benefit to instead generating a query string in Python and simply sending a single query instead:

query = f'''

#standardSQL
DECLARE FIELDS_TO_CHECK ARRAY<STRING>;
DECLARE i INT64 DEFAULT 0;

-- This fields would be inserted into the string using Python
SET FIELDS_TO_CHECK =  {['field1', 'field2', 'field3' ]}; 

LOOP
  SET i = i + 1;
  IF i > ARRAY_LENGTH(FIELDS_TO_CHECK) THEN 
    LEAVE; 
  END IF;
  EXECUTE IMMEDIATE '''
    INSERT result
    SELECT "''' || FIELDS_TO_CHECK[ORDINAL(i)] || '''", COUNT(''' || FIELDS_TO_CHECK[ORDINAL(i)] || ''') / COUNT(*) FROM `table_name`
  ''';

END LOOP; 

SELECT * FROM result; 

''' Are there any performance benefits to this method instead, or would it create multiple queries in the background anyway?

0 Answers
Related