Uploading CSV files as MongoDB documents using node.js

Admin 1
Viewed 2408

I am trying to save multiple CSV files into different MongoDB collections in JSON format using node.js express and mongoose, and then compare these files, currently, I managed to upload the CSV files to the MongoDB database but as documents, so every time I upload a new one it just gets added along with the other documents. which will make comparing the CSV files impossible cause you can't tell which documents will belong to what file.

A solution that seemed logical to me was to upload the CSV files as separate Collections and then compare the collections is this the best possible solution if so how could I implement that.

The mongoose model used :

const mongoose = require("mongoose");

const csvSchema = new mongoose.Schema({
  Sr: {
    type: String,
    required: true,
  },
  Cheque_No: {
    type: String,
    required: true,
  },
  Amount: {
    type: String,
    required: true,
  },
});

module.exports = mongoose.model("csvRecords", csvSchema);

The node.js snippet used to insert the CSV file in JSON format :

app.post("/upload-csv", uploads.single("csv"), (req, res) => {
  //convert csvfile to jsonArray
  csv()
    .fromFile(req.file.path)
    .then((jsonObj) => {
      //insertmany is used to save bulk data in database.
      //saving the data in collection(table)
      sheetModel.insertMany(jsonObj, (err, data) => {
        if (err) {
          res.status(400).json({
            message: "Something went wrong!",
          });
        } else {
          res.status(200).json({
            message: "File Uploaded Successfully!",
            result: data,
          });
        }
      });
    });
});
mongoose node.js mongodb csv
1 Answers

I think the problem you have right now is that you don't have an identifier in the document which can help you differentiate which two files you're comparing.

Since your use case is to just compare two documents(json) in a collection using a filename (unique identifier).

Sr Cheque_No Amount
1 3141341243 1234
2 3141341244 4000
3 3141341245 2000

If your csv file looks like above

You should model data in this way

{
    "filename": "xyz...", //save from the filename you're importing
    "csvData": [
        {
            Sr: 1,
            Cheque_No: 3141341243,
            Amount: 1234,
        },
        {
            Sr: 2,
            Cheque_No: 3141341244,
            Amount: 4000,
        },
        {
            Sr: 3,
            Cheque_No: 3141341244,
            Amount: 2000,
        },
        ...
    ]
}

Then you can query mongodb collection using filename as a query parameter, and write a custom logic or use a 3rd party library to compare two json documents (csvData)

const mongoose = require("mongoose");

const rowSchema = new mongoose.Schema({
  Sr: {
    type: String,
    required: true,
  },
  Cheque_No: {
    type: String,
    required: true,
  },
  Amount: {
    type: String,
    required: true,
  },
});

const csvSchema = new mongoose.Schema({
  fileName: { type: String, required: true },
  csvData: { type: [rowSchema], required: true },
});

module.exports = mongoose.model("csvRecords", csvSchema);


app.post("/upload-csv", uploads.single("csv"), (req, res) => {
  //convert csvfile to jsonArray
  const fileName = req.body.fileName
 
  csv()
    .fromFile(req.file.path)
    .then((jsonObj) => {
      //finding the document using fileName and setting csvData as the jsonObj
      sheetModel.findOneAndUpdate({ fileName: fileName }, {$set: { csvData: jsonObj, fileName: fileName}, { upsert: true }}, (err, data) => {
        if (err) {
          res.status(400).json({
            message: "Something went wrong!",
          });
        } else {
          res.status(200).json({
            message: "File Uploaded Successfully!",
            result: data,
          });
        }
      });
    });
});

Please check for syntax, implementation would be something similar like this.

user_40164550
Related