Azure Functions with Transfer-Encoding set to chunked

user_1721251 0
Viewed 537

I have an azure funciton (https://github.com/MCKRUZ/CSVXLS2JSON) that is supposed to take in a csv, xls, or zip file and translate that document to json. Now the problem that I'm running into is two fold

First, if I try to upload a file that is over 100 MB it tells me that the file is too big. I have gotten around this by allowing you to upload a zip file, which the function will unzip and then process. However, the resulting JSON payload going out is too big. I "thought" to get around this by allowing Transfer-Encoding set to chunked but I couldn't find ANY documentation online as to how to accomplish this.

Any thoughts?

azure-functions
1 Answers

More of a design question I guess:

  1. You could accept a URL as input instead of file content. That way you're not restricted by payload size. URL could be any URL pointing to datalake/external source with/without authentication tokens.
  2. Return results as compressed as well. Not a C# expert but most decent languages would allow you to create a compressed input/output stream. Read from compressed inputs stream, convert to json and write to compressed output stream. Return the contents of output stream at end.
  3. Return result as a URL. Write the output you have somewhere (like datalake) and return a URL to it. With/without auth.
user_4962890
Related