Iterate over large dataset from snowflake table in application using gosnowflake

Viewed 157

I am using gosnowflake to iterate over all rows of a large table in Snowflake ~10-50Million rows using below code.

db, err := sql.Open("snowflake", dsn)
defer db.Close()
query = `SELECT * from TEST_TABLE`
rows, err := db.QueryContext(ctx, query)
defer rows.Close()
for rows.Next() {
    err := rows.Scan(...)
    if err != nil {
        log.Fatalf("failed to get result. err: %v", err)
    }
}

For larger tables, above code ends up causing Out of Memory exception. Using pprof tool, it seems like most of the memory is used by gosnowflake. Below is one example pprof analysis image showing usage of ~5-6GB memory for a sample table with ~10M rows.

enter image description here

Memory usage keeps increasing linearly based on total number of rows in the source snowflake table.

Is there a way to iterate over large dataset tables while not exceeding configured memory limit like 2 GB other than pagination which will be multiple queries and will be costly?

0 Answers
Related