Copy into snowflake table from raw data file using Perl DBI

Viewed 223

There's not much info out there for perl dbi and snowflake so I'll give this a shot. I have a raw file, of which the headers are contained in line 1. This exact 'copy into' command works from the snowflake gui. I'm not sure if I can just take this exact command and put it into a perl prepare and execute.

COPY INTO DBTABLE.LND_LND_STANDARD_DATA FROM (
SELECT SPLIT_PART(METADATA$FILENAME,'/',4) as SEAT_ID,
 $1:auction_id_64 as AUCTION_ID_64,
 DATEADD(S,\$1:date_time,'1970-01-01') as DATE_TIME,
 $1:user_tz_offset as USER_TZ_OFFSET,
 $1:creative_width as CREATIVE_WIDTH,
 $1:creative_height as CREATIVE_HEIGHT,
 $1:media_type as MEDIA_TYPE,
 $1:fold_position as FOLD_POSITION,
 $1:event_type as EVENT_TYPE
FROM @DBTABLE.lnd.S3_STAGE_READY/pr/data/standard/data_dt=20200825/00/STANDARD_FILE.gz.parquet)
pattern = '.*.parquet'  file_format = (TYPE = 'PARQUET' SNAPPY_COMPRESSION = TRUE)
ON_ERROR = 'SKIP_FILE_10%'
my $SQL = "COPY INTO DBTABLE.LND_LND_STANDARD_DATA FROM (
SELECT SPLIT_PART(METADATA\$FILENAME,'/',4) as SEAT_ID,
 \$1:auction_id_64 as AUCTION_ID_64,
 DATEADD(S,\$1:date_time,'1970-01-01') as DATE_TIME,
 \$1:user_tz_offset as USER_TZ_OFFSET,
 \$1:creative_width as CREATIVE_WIDTH,
 \$1:creative_height as CREATIVE_HEIGHT,
 \$1:media_type as MEDIA_TYPE,
 \$1:fold_position as FOLD_POSITION,
 \$1:event_type as EVENT_TYPE
FROM \@DBTABLE.lnd.S3_STAGE_READY/pr/data/standard/data_dt=20200825/00/STANDARD_FILE.gz.parquet)
pattern = '.*.parquet'  file_format = (TYPE = 'PARQUET' SNAPPY_COMPRESSION = TRUE)
ON_ERROR = 'SKIP_FILE_10%'";
my $sth = $dbh->prepare($sql);
$sth->execute;

In looking at the output from snowflake I see this error

syntax error line 3 at position 4 unexpected '?'. syntax error line 4 at position 13 unexpected '?'.

COPY INTO DBTABLE.LND_LND_STANDARD_DATA FROM (
SELECT SPLIT_PART(METADATA$FILENAME,'/',4) as SEAT_ID,
$1? as AUCTION_ID_64,
DATEADD(S,$1?,'1970-01-01') as DATE_TIME,
$1? as USER_TZ_OFFSET,
$1? as CREATIVE_WIDTH,
$1? as CREATIVE_HEIGHT,
$1? as MEDIA_TYPE

Do I need to create bind variables for each of the columns? I usually pull in the data from the file and put them into variables but this is different as I can't read the raw file first, it has to come directly from the copy into command.

Any help would be appreciated.

1 Answers

It was interpreting the : as a bind variable value, rather than a value in a variant. I used the bracket notation, instead like the following:

my $SQL = "COPY INTO DBTABLE.LND_LND_STANDARD_DATA FROM (
SELECT SPLIT_PART(METADATA\$FILENAME,'/',4) as SEAT_ID,
 \$1['auction_id_64'] as AUCTION_ID_64,
 DATEADD(S,\$1['date_time,'1970-01-01') as DATE_TIME,
 \$1['user_tz_offset'] as USER_TZ_OFFSET,
 \$1:creative_width'] as CREATIVE_WIDTH,
etc...

That worked

Related