More scalable way to write this Node.js logic for retrieving and updating CSV data from Google Cloud Storage?

user_5779866 0
Viewed 292

I'm creating my first Node.js REST API test project, which retrieves prizes. The logic is meant to do the following:

  • Retrieve a CSV from the Google Cloud Storage bucket associated with the project
  • Parse the CSV
  • Find the first row where the column "Claimed" isn't populated
  • Update the "Claimed" column to "claimed!"
  • Overwrite the data in the CSV file in Google Cloud Storage
  • Return the prize name associated with that row

The logic I have is currently working locally, but I'm wondering if there is a better, more scalable way to write the code. I'll be testing it at a 500/min rate limit for around 100k users per day and would eventually pass an external user-id to be stored in the "Claimed" column. There will be a total of 500k rows (prizes) in the CSV.

The code I'm using is below. Any suggestions for making it scalable would be much appreciated! Thank you in advance.

const csv = require('csv-parser');
const fs = require('fs');
const jsonexport = require('jsonexport');

const bucketName = 'MY-BUCKET';
const filename = 'MY-CSV';

const {Storage} = require('@google-cloud/storage');

const storage = new Storage({keyFilename: "MY-KEY.json"});

const myBucket = storage.bucket(bucketName);
const file = myBucket.file(filename);

let dataArray = [];

file.createReadStream()
  .pipe(csv())
  .on('data', function (data) {
    dataArray.push(data);
  })
  .on('end', function(){
    let prize = dataArray.find(element => element.Claimed == "");
    prizeName = prize.Prize_name;
    prize.Claimed = "claimed!";
    jsonexport(dataArray,function(err, transformedData){
      if(err) return console.log(err);
      file.save(transformedData, function(err) {
        if(err) return console.log(err);
      });
    });
    return prizeName
});
google-cloud-storage node.js csv rest
1 Answers

Okay, I have my experience with that. According to a question, it's more about performance, and I suggest that basic code works fine.

I guess the bottleneck is fs and createReadStream. It works fine, but it's not async (this question shows us why).

actually it's, but... you'll flood your RAM with your file, if you would like to store it at once

const fs = require('fs');
const readFile = promisify(fs.readFile);
let file = await readFile(`${path_}/${file}`, {encoding: 'utf8'});

So even if you have 10 files with 50k users instead of one with 500k.. you will needed to parse all 10 at once, but while you do it synchronous..

Well, you logic won't work x10 faster as you may think.

By the way, if you have another code that does the same functionality, but you don't know how to measure it performance speed, use this:

console.time('benchmark_name')
/* Some action here */
console.timeEnd('benchmark_name')

I understand that my answer is not an answer at all, but this is how I would solve this problem as if I were you:

  • drop the CSV - it will only generate a problem for you, especially when you are dealing with 100K+ rows.
  • try cloud infrastructure: if you need to store your data somewhere, use MongoDB Atlas (free tier) for example. And don't forget too add some indexes on relevant columns. So as for now, you don't need fs and it streams at all.

Focusing on Mongo Atlas in this case is just an option, you could use Azure Cosmos DB, or anything else. Even use AWS / GCP free tier VPS and launch any suitable-your-needs database. The point is, — run away from Google Sheets/Drive.

Why you should avoid CSV / Google Drive ?

They are not so bad, as you may think but ask yourself, if google drive / csv is the efficient way to store data, why people using databases instead of storing all info in one big *.csv file? I guess this metaphor make sense.

So back to our future DB..

As for now, you only need to connect to your database and modify it's values. You modify everything what you need at once, via one query, instead of:

let prize = dataArray.find(element => element.Claimed == "");

You do not need to find every row one-by-one. This is the precious scalable what you was asking about.

The only thing that you'll need is: AWS Lambda, MongoDB Stitch/Realm, webhook, API to modify you data in DB or add to your form (if you data should been updated via http form). By the way, if you are scared and not-yet-ready to jump and leave Google Sheets behind, you could read this article first and try to connect Google Sheets and your DB (in this case it's MongoDB, but Google Sheets has support of Google Script, so connect any other DB won't be a big problem)

So all this steps will help your app to be more scalable, as you have requested. Also using DB will solve any-other problem with data-validation and so on..

As an opposite site, in one of my projects I am having a dependence from a data source which is actually posts a big *.csv sheet. How big is it? 65K+ rows, and find and updating every value in it, costs 7-12 mins of resourceful time. God, I how I hate that guy, who using csv instead of API endpoints / DB connectors.

user_74756150
Related