How to skip lines based on a criteria while loading a CSV file?

user_12764749 0
Viewed 145

I'm trying to load a csv file with D3.js but I want to skip some lines based on some criteria When I apply filter() it says no such function. Any help would be appreciated.

var dataSet = d3.csv("mydata.csv", function(d) 
      {
       // trying to skip the line based on the below condition
       if (d.year < 1900)
       {
          //skip loading, not sure what should I return
          return false;
        }
       else
       return {
         
          name: d.name,
          year: d.year,
          average_rating: d.average_rating,
          user_rated: d.user_rated
       };
     })

data.csv:

name,year,average_rating,users_rated
King of Tokyo,2011,7.23048,48611
Love Letter,2012,7.25253,47014
d3.js javascript
2 Answers

There is no way to skip lines when loading the CSV. So, if your CSV has 1MB but only a couple of lines fit the criteria, you'll still have to load the whole 1MB. However, despite you having "skip loading" in your question it seems to me that you just want to filter the parsed CSV. If that's correct, you can either use a regular filter or, as you're trying to do in your question, you can use a row conversion function.

In that case just check the year: you return the whole object if it fits the criterion or, otherwise, return null. Check this simple demo:

const csv = `name,year
foo,1300
bar,1800,
baz,2200`;

const data = d3.csvParse(csv, d => +d.year < 1900 ? d : null);

console.log(data)
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min.js"></script>

Pay attention to the fact that numbers are parsed as strings, hence the +d.year.

user_57689080

Worked just fine as:

const dataSet = d3.csv("mydata.csv", function(d) 
          {
           // skips all the lines where d.year >= 1900 condition
           if (d.year >= 1900){
              return {             
              name: d.name,
              year: d.year,
              average_rating: d.average_rating,
              user_rated: d.user_rated
               }
            };
         }).then(function(d){

          //some more data manipulation
          // d.field = +d.field;
          
         })
         .catch(function(error) 
         {
            console.log(error);
         });
user_114383960
Related