How to speed up an endpoint that's passing a significant amount of data to db

user_13359923 0
Viewed 69

Essentially I have a React web app that I'm creating to upload a large set of data to a db. As of now I'm using a test file that's only around 3000 lines long and the total time to upload all the data from the file is around 4 minutes. I need to speed this up significantly but I'm not totally sure how. For reference the actual size of files I plan to upload will be around 100k lines long. Also another issue that I'm pretty sure I can fix with a little tinkering is I start to run out of net resources around 80k request so if there are any suggestions to modifying my delayQue function to avoid that then it would be appreciated as well.

Test file size: 76kb

Real file size: 5 mb

Endpoint

  const stepData = async() =>{

    const body = {a,b,c,d,e,f}
    assignToBody(body)

    try{

        const response = await fetch('http://localhost:5000/stepdata',{
                method:"POST",
                headers: {"Content-Type": "application/json"},
                body: JSON.stringify(body)
    })}catch(err){
        console.log(err.message)
    }
    
}

Throttle (note: assignVals just assigns the values to the body in my endpoint)

    const delayQue = () =>{

        assignVals(row)
        row++
        
        if(row % 500 == 0){
            setTimeout(delayQue,2000)
        }else{
            setTimeout(delayQue,100)
        }
    
  }

also another thing incase its important each line contains anywhere from 6 to 20 points of data. I lowered my number of calls by assigning the values to an object and passing the object. Any help on speeding this up will be greatly appreciated.

Edit: To clarify the issue is that I am simply having to make 100k request or more and am trying to find a way to speed up my request or at least manage them better instead of using settimeout and embedded try and catches.

Update: I'm fairly certain the issue is a resouce leak in my endpoint

reactjs node.js endpoint javascript
1 Answers

Regarding the amount of data transferred over the network:

File size can be reduced very efficiently using a compression algorithm like gzip or brotli. Depending on the webserver you are using you will have to activate compression first for it to be used.

Here a example how to use with express: https://expressjs.com/en/resources/middleware/compression.html

Depending on the data you want to transfer, you can reduce the file size significantly. Repetitive text files compress best. Already compressed files like jpeg images don't compress at all.

user_17540760
Related