Algorithm run from within Node HTTP request takes much longer to run

Viewed 350

I have a node app which plots data on an x,y dot plot graph. Currently, I make a GET request from the front end and my back end node server accepts the requests, loops through an array of data points, draws a canvas using Node Canvas and streams it back to the front end where it's displayed as a PNG image.

Complicating things is that there are can be polygons so my algorithm calculates if a point is inside a polygon, using the point in polygon package, and colors that data point differently if it is.

This works fine when there are less than 50,000 data points. However, when there are 800,000 the request takes approximately 23 seconds. I have profiled the code and most of that time is spent looping through all the data points and figuring out where to plot it on the canvas and what color (depending on if it's in one or more polygons). Here's a plunker i made. Basically i do something like this:

for (var i = 0; i < data.length; i++) {

  // get raw points
  x = data[i][0];
  y = data[i][1];

  // convert to a point on canvas
  pointX = getPointOnCanvas(x);
  pointY = getPointOnCanvas(y, 'y');

  color = getColorOfCell(pointX, pointY);

  color = color;

  plotColor.push({
      color: color,
      pointX: pointX,
      pointY : pointY
  });

}

// draw the dots down here

The algorithm itself is not the problem. The issue I have is that when the algorithm is run within a HTTP request, it takes a long time to calculate what color a point is - about 16 seconds. But if do it in chrome on the front end, it takes just over a second (see the plunker). When I run the algorithm on the command line with Node, it takes less than a second. So the fact that my app runs the algorithm within a HTTP request is slowing it down massively. So couple of questions:

Why would this be? Why does running an algorithm from within a HTTP request take so much longer?

What can I do to fix this, if anything? Would it somehow be possible to make a request to start the task, and then notify frontend when finished and retrieve the PNG?

EDIT I fully tested running the algorithm and creating a PNG through the command line. It's much quicker, less than half a second to work out what color each of the 800k data points should be. Im thinking of using socket to make a request to the server and start the task, then have it return the image. I'm baffled though why the code should take so long when run within a HTTP request...

EDIT The problem is Mongo and Mongoose. I store the coordinates of each polygon in Mongo. I fetch these coordinates once but when I compare them to each x, y point/. Somehow, this is what's massively delaying the algoritm. If I close the Mongo document, the algorithm goes from 16 seconds to 1.5 seconds......

Edit @DevDig pointed out the main problem in the comments section - when using a Mongoose object there are lots of getters and setters slowing it down. Using lean() in the query reduces algorithm from 16 seconds to 1.5 seconds

1 Answers
Related