I'm using Python to:
- Fetch a list of values from column in table A
- Iterate over the list and query table B for each value
This seems like an awful pattern, since BigQuery isn't transactional. I know we can loop in BigQuery. But I'm not sure if there is any benefit to instead generating a query string in Python and simply sending a single query instead:
query = f'''
#standardSQL
DECLARE FIELDS_TO_CHECK ARRAY<STRING>;
DECLARE i INT64 DEFAULT 0;
-- This fields would be inserted into the string using Python
SET FIELDS_TO_CHECK = {['field1', 'field2', 'field3' ]};
LOOP
SET i = i + 1;
IF i > ARRAY_LENGTH(FIELDS_TO_CHECK) THEN
LEAVE;
END IF;
EXECUTE IMMEDIATE '''
INSERT result
SELECT "''' || FIELDS_TO_CHECK[ORDINAL(i)] || '''", COUNT(''' || FIELDS_TO_CHECK[ORDINAL(i)] || ''') / COUNT(*) FROM `table_name`
''';
END LOOP;
SELECT * FROM result;
''' Are there any performance benefits to this method instead, or would it create multiple queries in the background anyway?