I have created a Snowflake external stage on AWS S3 bucket. Also, I have setup a Snowpipe which executes the copy command when there is a file PUT on the S3 bucket. The Snowpipe gets notification from an SNS notification. The pipe has been configured to abort on error. Now, if there is any error while copying files from S3 into Snowflake ...say due to data mismatch... the Snowpipe simply aborts. Is there a way I get notified when the pipe fails? Because, I do not know if I am loosing any file from getting copied due to error.
3 Answers
Snowflake does not currently have notification/alerting services available for anything other than RESOURCE MONITORs, which are for warehouse credit consumption. The best way to monitor issues with Snowpipe is to leverage a third party reporting/monitoring tool that queries the Snowflake views.
I would recommend reviewing the following github project, as well, which is an open-source monitoring/alerting tool written for Snowflake:
No, Currently snow pipe won't send any notification on failure. Its recommended to review the file status in snow pipe queue or use any third part tools to monitor your snowpipe queue.
Related