send data from stage to multi column table in snowflake

user_12320257 0
Viewed 179

I have an internal named stage where json files are stored and from there I want to store them in snowflake table. The structure of destination table is as follows,

file_name (string)
load_date (timestamp)
data      (variant)

I am using the following query to move the data from stage to table

copy into tableName (data) from @stagename/filename.json;

But the above query is only populating the data column, what I want is to insert the timestamp and filename too. Any idea what changes I need to make in the query? Thanks

snowflake-schema snowflake-cloud-data-platform sql
1 Answers

You need to use a COPY statement with a transformation - documentation here. When you use that method you can query the metadata of the files to get the filename, row number etc - documentation for that here.

Example file filename.json uploaded to an internal stage called stagename:

[{"name": "simon"},{"name": "jason"}, {"name": "jessica"}]

Sql to load create and load table:

-- Create example table first with 3 columns
create or replace transient table test_table
(
    file_name varchar,
    load_date timestamp,
    data      variant
);


-- Load with transformation: 
copy into test_table (file_name, load_date, data) from (
    select
        metadata$filename,
        current_timestamp,
        f.$1
    from @stagename/filename.json f
)
    file_format = (
        type = json
            strip_outer_array = true
        )
    force=true
;

Results:

+-------------+-----------------------------+-----------------------+
|FILE_NAME    |LOAD_DATE                    |DATA                   |
+-------------+-----------------------------+-----------------------+
|filename.json|2021-07-16 08:56:24.075000000|{"name": "simon"}      |
|filename.json|2021-07-16 08:56:24.075000000|{"name": "jason"}      |
|filename.json|2021-07-16 08:56:24.075000000|{"name": "jessica"}    |
+-------------+-----------------------------+-----------------------+
user_32872500
Related