Comparing three large very large arrays and creating a new object

user_8342740 0
Viewed 163

Alright, so I have three arrays with about 65k objects in each in node.js. They all share an ID - PARCELID, SBL, and SBL20 are the same. I want to combine the objects from the different arrays into a single object that is then pushed into the final array. For whatever reason, the output I get contains 130k+ objects. This is also pretty inefficient so if there is a better way to do this, I am all ears - I have attempted to use map() though I was only able to compare two arrays, not three. I have a fourth I would like to add into the mix as well.

var final = new Array();
  count=0
  TaxParcels.forEach((TaxParcel) => {
    TaxBills.forEach((TaxBill) => {
      if (TaxParcel.PARCELID == TaxBill.SBL20) {
        CodeEnforcements.forEach((CodeEnforcement) => {
          if (TaxParcel.PARCELID == CodeEnforcement.SBL) {
            parcel = {
              ID: TaxParcel.PARCELID,
              DETAILS: {
                TaxParcel: TaxParcel,
                TaxBill: TaxBill,
                CodeEnforcement: CodeEnforcement,
              },
            };
            final.push(parcel);
            count++
          }
        });
      }
    });
  });
  console.log(final);
  console.log(count)

node.js compare foreach arrays javascript
2 Answers

You can get performance gain by first creating a Map, keyed by the keys, with empty objects related to each. Then inject the objects into those objects, and filter the result to only have objects with all 3 keys:

// sample data
let TaxParcels = [{ PARCELID: 3 }, { PARCELID: 9 }, { PARCELID: 7 }];
let TaxBills = [{ SBL20: 9 }, { SBL20: 1 }, { SBL20: 3 }];
let CodeEnforcements = [{ SBL: 3 }, { SBL: 9 }, { SBL: 1 }];

// Solution
let map = new Map([...TaxParcels, ...TaxBills, ...CodeEnforcements].map(o => 
    [o.PARCELID??o.SBL20??o.SBL, {}])
);

for (let o of TaxParcels) map.get(o.PARCELID).TaxParcel = o;
for (let o of TaxBills) map.get(o.SBL20).TaxBill = o;
for (let o of CodeEnforcements) map.get(o.SBL).CodeEnforcement = o;

let result = [...map.values()].filter(o => Object.keys(o).length === 3);
console.log(result);
console.log(result.length);

user_54598390

The simplest (and maybe not most significant) way is to avoid forEach here: it iterates over the whole arrays pointlessly even after the element is found. There is no way to break forEach. It may be replaced with for-of loops with break or with find array methods. This will also eliminate duplicates. The last way example:

const TaxParcels = [{ PARCELID: 1}, { PARCELID: 2}, { PARCELID: 3}];
const TaxBills = [{ SBL20: 1}, { SBL20: 2}, { SBL20: 3}];
const CodeEnforcements = [{ SBL: 1}, { SBL: 2}, { SBL: 3}];

const final = [];
let count=0;

for (const TaxParcel of TaxParcels) {
  const ID = TaxParcel.PARCELID;

  const TaxBill = TaxBills.find(({ SBL20 }) => SBL20 === ID);
  const CodeEnforcement = CodeEnforcements.find(({ SBL }) => SBL === ID);

  if (TaxBill && CodeEnforcement) {
    count++;
    final.push({ ID, DETAILS: { TaxParcel, TaxBill, CodeEnforcement } });
  }
}

console.log(count);
console.log(JSON.stringify(final, null, '  '));

But maybe the most significant improvement is to use more appropriate id-keyed structures as in other answer here.

user_26255600
Related