Merging arrays of objects from different CSVs

Question

I am using D3JS to visualize data collected by Jenkins. The only problem is that I want to visualize multiple lines at the same time and the Jenkins scripts creating the CSV files for these are creating multiple separate files. For example:

File1:

ID;ValueA;ValueB
ID01;10;10
ID02;30;40

File2:

ID;ValueC;ValueD
ID01;20;30
ID03;50;60

This way the final result should be something like this after merging:

ID;ValueA;ValueB;ValueC;ValueD
ID01;10;10;20;30
ID02;30;40;;
ID03;;;50;60

My problem was that concat simply does the following:

ID01;10;10
ID02;30;40
ID01;20;30
ID03;50;60

user_5768908 · Answer

There are several ways to merge those arrays of objects, which can favour speed, or brevity, or readability etc...

This is one way to do it, out of several. First, let's get our parsed CSVs. Since I cannot use a real CSV in the StackOverflow snippet I'll use a template literal; also, since you have semicolons in your CSV, I'll use a DSV parser:

const file1 = `ID;ValueA;ValueB
ID01;10;10
ID02;30;40`;

const file2 = `ID;ValueC;ValueD
ID01;20;30
ID03;50;60`;

const csv1 = d3.dsvFormat(";").parse(file1);
const csv2 = d3.dsvFormat(";").parse(file2);

Now we get the headers, based on the CSVs:

const headers = [...new Set(csv1.columns.concat(csv2.columns))].filter(function(d) {
  return d !== "ID"
});

Finally comes the function:

const mergedData = [];

csv1.concat(csv2).forEach(function(row) {
  const foundObject = mergedData.find(function(d) {
    return d.ID === row.ID
  });
  if (foundObject) {
    headers.forEach(function(d) {
      if (row[d]) foundObject[d] = row[d];
    });
  } else {
    headers.forEach(function(d) {
      if (!row[d]) row[d] = "";
    });
    mergedData.push(row)
  };
});

We first create an empty array for the merged data. Then, we search in that array an object that matches the ID for each object in the two CSVs: if it's found, the extra columns are created; if it's not found the original row is pushed, with "" as the value for the columns that it doesn't have, which is exactly what d3.csv() does.

Here is the demo:

const file1 = `ID;ValueA;ValueB
ID01;10;10
ID02;30;40`;

const file2 = `ID;ValueC;ValueD
ID01;20;30
ID03;50;60`;

const csv1 = d3.dsvFormat(";").parse(file1);
const csv2 = d3.dsvFormat(";").parse(file2);

const headers = [...new Set(csv1.columns.concat(csv2.columns))].filter(function(d) {
  return d !== "ID"
});

const mergedData = [];

csv1.concat(csv2).forEach(function(row) {
  const foundObject = mergedData.find(function(d) {
    return d.ID === row.ID
  });
  if (foundObject) {
    headers.forEach(function(d) {
      if (row[d]) foundObject[d] = row[d];
    });
  } else {
    headers.forEach(function(d) {
      if (!row[d]) row[d] = "";
    });
    mergedData.push(row)
  };
});

console.log(mergedData);

Merging arrays of objects from different CSVs

1 Answers