node.js - use values from a .csv file in an http get request (API)

user_7446623 0
Viewed 639

I have been researching my question, and the more I research the more confused I get.

I need to use node.js to merge information from an API into an existing .csv file and output a new .csv file. The existing .csv file has four columns, including an accountId. The data returned from the API will have an accountId column plus two additional columns that are not in the existing .csv file. The new .csv file needs to have the four existing columns plus the two new columns.

Here is my code so far:

console.log('Running merge');

const csv = require('csv-parser');
const fs = require('fs');
const json2csv = require('json2csv');
const axios = require('axios');

let dataArray = [];

fs.createReadStream('input - input.csv')
    .pipe(csv())
    .on('data', function (data) {
        // data.employee = 'employee';
        // console.log("data: " + data);
        // var accountId = data.accountId;
        // console.log("accountId = " + accountId);
    
        dataArray.push(data);
    })
    .on('end', function() {
        // console.log('data.accountId: ' + data.accountId);
        var result = json2csv.parse({ data: dataArray, fields: Object.keys(dataArray[0]) });
        fs.writeFileSync('employeeFile.csv', result);
});

The first thing I did when writing the code was add the data.employee = "employee" line to verify that when I wrote the new .csv file the new column would be in it. That worked. My next goal, and where I am running into trouble, is retrieving the accountId so I can use it in the callout.

I had expected to be able to do the following:

.on('data', function (data) {
    var accountId = data.accountId;
    // console.log("accountId = " + accountId);
    axios.get('https://company.callout.io/v1/accounts/' + accountId)
        .then(function (response) {
            // figure out what I need to do from there
        }

But when I do the console.log, I get accountId printed out with a value of undefined for each one.

I did read that this might be due to the asynchronous nature of JavaScript, and I thought maybe I should put the callout in the .on(end) par of the createReadStream function. So I tried the console.log you see with data.accountId, but I received an error that data is not defined. My mistake was that the data is now in the dataArray. I've been trying to figure out how to retrieve the values for accountId from the array, but I'm stuck. I've tried the following:

.on('end', function() {
    let accountId = 'accountId';
    for(accountId in dataArray) {
        if (dataArray.hasOwnProperty(accountId)) {
            console.log('Account Id is ' + dataArray[accountId]);
        }
    }

But all I get is 7 lines of Account Id is [object Object]

I've also tried:

.on('end', function() {
    let accountIdArray = dataArray.map(accountId => {
        return dataArray['accountId']; //tried accountId with and without quotes
    });
    console.log('Account Ids are ' + accountIdArray);

But I just ended up with Account Ids are ,,,,,,

I am just really stuck at this point. Any help would be greatly appreciated.

EDIT

Here is the sample .csv file I have been given to work with:

Account ID,Account Name,First Name,Created On
12345,lexcorp,Lex,1/12/2011
8172,latveriaembassy,Victor,11/19/2014
1924,brotherhood,Max,2/29/2012
222222,leagueofassassins,Ra's,3/1/2012
48213,kingpin,Wilson,7/7/2015
918299,oscorp,Norman,4/29/2014
88888,dococt,Otto,8/8/2013

Here is a sample of the data returned from the service:

{
    "account_id": 12345,
    "status": "good",
    "created_on": "2011-01-12"
}

The "created_on" in the return of service must be under a column called Status Set On, since there is already a Created On column in the existing .csv file.

Please note that the .csv file I was given was simply a 'happy' file. My code will be run against a lot of .csv files that may have negative account numbers, missing values, and other things that I have to catch with error coding. The point is, I can't 'hard code' anything based on what is in the file I was given.

node.js csv arrays javascript
1 Answers

If you change the code like so, using bracket notation to access the Account ID, you should get the desired result.

fs.createReadStream('input - input.csv')
    .pipe(csv())
    .on('data', function (data) {
        console.log("data: ", data);
        var accountId = data['Account ID'];
        console.log("accountId = " + accountId);
        dataArray.push(data);
    })
    .on('end', function() {
        let accountIdArray = dataArray.map(account => account['Account ID'])
        console.log('Account Ids are ' + accountIdArray);
    });

Once you have the account ID in a separate variable, either in the .on('data', callback or the .on('end', callback you should be able to make the axios calls as necessary.

user_72372240
Related