How to parse a csv into a dataTable quickly and efficiently using D3 v6

user_6474694 0
Viewed 496

What I'm trying to do

I'm trying to load a CSV with 65 thousand rows into a dataTable. Trying to make a static webpage to showcase some data I just parsed,

I'm using the following libraries

<script src="https://code.jquery.com/jquery-3.5.1.js" charset="utf-8"></script>
<script src="https://cdn.datatables.net/1.10.23/js/jquery.dataTables.min.js" charset="utf-8"> </script>
<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/dt/dt-1.10.23/datatables.min.css" />

What I did

Here is my code on the javascript side. My HTML just has a body and head. The head is then populated with a table

var tabulate = function (data, columns) {
var table = d3.select('body').append('table').attr("id", "example").attr("class", "display nowrap")
var thead = table.append('thead')
var tbody = table.append('tbody')

thead.append('tr')
    .selectAll('th')
    .data(columns)
    .enter()
    .append('th')
    .text(function (d) {
        return d
    })

var rows = tbody.selectAll('tr')
    .data(data)
    .enter()
    .append('tr')

var cells = rows.selectAll('td')
    .data(function (row) {
        return columns.map(function (column) {
            return {
                column: column,
                value: row[column]
            }
        })
    })
    .enter()
    .append('td')
    .text(function (d) {
        return d.value
    })

return table;
}
d3.csv("all_engineers.csv", function (data) {
    var columns = d3.keys(data[0]);
    tabulate(data, columns);
    $('#example').DataTable();
})

What is the problem?

It's tooooooo slow. It could be due to the big data set. I was wondering if there was an easy fix. I know D3 is super efficient and there might be something I'm not leveraging.

I also can't seem to use this code with d3 v6. I ended up using d3 v3 instead because this is the example I found that worked with datatables.

Thank you in advance for taking the time.

Update

Thanks to altocumulus We figured out how to use d3 v6

The old implementation (using D3 V3) jsfiddle.net/2nwasz43

the Updated Implementation (using D3 V6) jsfiddle.net/gndv6rq0/1

d3.js datatables
1 Answers

Loading time before was taking more than 2 minutes before. I removed a few columns from the CSV and decided to add a loading gif to the HTML page since making this instantaneous was out of the question in terms of what I wanted to do

What really helped reduce the load time was loading the d3 data object directly into the datable instead of putting it in the DOM. I was getting page unresponsive in the latter but that seemed to have gone away in the former.

I'm using d3 version 3 instead of version 6 since the syntax seems cleaner and more familiar to me. If you want to use d3 version 6 then you would need to watch out for the API differences (look at https://jsfiddle.net/gndv6rq0/1 for reference)

What I ended up with was the following script

function hideLoader() {
$('#loading').hide();
}

setTimeout(hideLoader, 80 * 1000);

var tabulate = function (columns) {
var table = d3.select('body').select('table')
var thead = table.append('thead')

thead.append('tr')
    .selectAll('th')
    .data(columns)
    .enter()
    .append('th')
    .text(function (d) {
        return d
        })

    return table;
}
d3.csv("All_engineers_reduced.csv", function (data) {
    var columns = d3.keys(data[0]);
    tabulate(columns);
    $('#example').DataTable({
        data: data,
        columns: [
        {
            "data": "Engineer_ID"
        },
        {
            "data": "Arabic_Names"
        },
        {
            "data": "Latin_Names"
        },
        {
            "data": "Field_ID"
        },
        {
            "data": "SubField_ID"
        },
        {
            "data": "Field"
        },
        {
            "data": "SubField"
        }

    ],
    processing: true,
        language: {
            processing: "<img src='https://media1.giphy.com/media/feN0YJbVs0fwA/giphy.gif'>"
        }
    });
    hideLoader();
})

The loaded libraries are still

    <script src="https://d3js.org/d3.v3.min.js" charset="utf-8"></script>
<script src="https://code.jquery.com/jquery-3.5.1.js" charset="utf-8"></script>
<script src="https://cdn.datatables.net/1.10.23/js/jquery.dataTables.min.js" charset="utf-8"></script>
<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/v/dt/dt-1.10.23/datatables.min.css" />

You can see the final result in the link below https://ebrahimkaram.github.io/LebaneseEngineers/

user_64746940
Related