I'm trying to do a performance comparison between CTEs (WITH) and Temporary Tables. I've converted the query to utilized Temp tables but am seeing a scenario where the execution fails when run as a whole because one of the early temporary table executions gets cancelled. SAMPLE:
// Produces large result set
CREATE temporary table tt1 as
SELECT DISTINCT
t1.c1
FROM t1
//ADDITIONAL LOGIC
;
CREATE temporary table tt2 as
SELECT t2.c1
t2.c2
FROM t2;
CREATE temporary table tt3 as
SELECT t1.c1
FROM t1
INNER JOIN tt1
on t1.c1 = tt1.c1;
The execution seems to get cancelled as tt3 begins to run, there is a dependency from tt3 back to tt1.
I have tried CTAS as well as defining the table with a standard CREATE TABLE then doing an INSERT INTO but it results in the same where the execution gets cancelled.
Is there a way to sequence these better via standard sql syntax without having to go the route of a stored procedure or task?
Edit[2] Here is a simpler scenario using the COVID dataset that behaves the same. Termination in both cases seems to be around 35 seconds. So for replication purposes if you can generate a query that runs for 2 minutes, then create a simple subsequent temporary table that selects from the first you should be able to replicate.
CREATE TEMPORARY TABLE PLATFORM_EVALUATIONS.PUBLIC.TT1 AS
select a.COUNTRY_REGION
, a.PROVINCE_STATE
, a.COUNTY
, current_date() c_date
, count(*) counts
from PUBLIC.JHU_COVID_19_TIMESERIES a
inner join public.jhu_dashboard_covid_19_global b
on a.country_region = b.country_region
where 1=1
and a.COUNTRY_REGION = 'United States'
group by a.COUNTRY_REGION
, a.PROVINCE_STATE
, a.COUNTY;
CREATE TEMPORARY TABLE PLATFORM_EVALUATIONS.PUBLIC.TT2 AS
SELECT COUNTRY_REGION
, COUNTS
FROM PLATFORM_EVALUATIONS.PUBLIC.TT1;
Edit1: Work History Screenshot You can see in the screen shot that _valid_barcodes (tt1) is cut short in duration (normally 2m 14s independently) and other steps begin prior to its completions/termination.
// this is tt1 in my example it produces 3,657,599 rows in 2m 14s but is cancelled short when run as a whole.
CREATE temporary table business_vault_dev.dnr._valid_barcodes as
SELECT DISTINCT
e.Barcode
FROM BUSINESS_VAULT_DEV.DNR.DIM_EVENT_VW e
INNER JOIN BUSINESS_VAULT_DEV.DNR.DIM_PACKAGE_VW p
on e.BARCODE = COALESCE(p.Barcode, p.Barcode)
INNER JOIN BUSINESS_VAULT_DEV.DNR.DIM_ORDER_VW o
on p.order_key = o.order_key
INNER JOIN BUSINESS_VAULT_DEV.SCORECARD.DIM_CUSTOMER_VW cl
on o.CUSTOMER_ID=cl.CUSTOMER_ID
WHERE e.Event_Type='Delivered'
and o.service_code not in ('XD', 'PU')
and lower(cl.NAME) IN ('a', 'b', 'c')
GROUP BY e.Barcode
HAVING date(min(e.EST_Event_DateTime)) between '2020-09-05' and '2020-10-02';
// this is tt2
CREATE temporary table business_vault_dev.dnr._driver_merge as
select
//redacted
from data_lake.driver.driver_table1
union
select
//redacted
from data_lake.driver.driver_table2
union
select
//redacted
from data_lake.driver.driver_table3
UNION
select
//redacted
from data_lake.driver.driver_table4;
// this is unreferenced in my example but completes successfully and references the previous temp table
CREATE temporary table business_vault_dev.dnr._driver as
select
// redacted
from business_vault_dev.dnr._driver_merge dm
group by dm.contractor_number;
// this is tt3 in my example
CREATE temporary table business_vault_dev.dnr._slef_dedup as
SELECT
e.Barcode
// redacting columns here
FROM BUSINESS_VAULT_DEV.DNR.DIM_EVENT_VW e
INNER JOIN business_vault_dev.dnr._valid_barcodes vb
ON e.Barcode = vb.Barcode
LEFT JOIN data_lake.driver.driver_table1 cn
ON //redacted
LEFT JOIN data_lake.driver.driver_table2 pcp
ON //redacted
;