Does Utilities.parseCsv has limitations over the input data size?

Question

I am trying to parse a CSV in Apps Script which has 52 columns and 126543 rows. Data size is 72 MB.

After parsing it using Utilities.parseCsv method, the size of the data is 95104 however the expectation is that it should be 126543 i.e. same as the initial data size.

After removing few columns from the CSV and again repeating the above step, the size of the data after parsing increased but it still did not match the initial data size.

Is there a size limitation with Utilities.parseCsv? I could not find any documentation referring to this limitation if it is true.

Kindly put some light on this and is there any alternative solution which does not have size constraints?

TheMaster · Answer

If it did have a size limitation, it would throw a error. The most probable reason for reduced rows is incorrectly formatted csv(improper double quotes)/corrupt csv. Try checking each row's length:

const csvChecker_ = csv => {
  const arr = Utilities.parseCsv(csv),
  len = arr[0].length;
  arr.forEach((row,i) => row.length === len ||
    console.error(`Incorrect length at row ${i+1}. Row parsed incorrectly: ${JSON.stringify(row)}`))
}

You can manually check that row to figure out the problem or build a custom csv parser for your csv.

Does Utilities.parseCsv has limitations over the input data size?

1 Answers

Related: