Download file from from GCP Storage bucket is very slow with npm library @google-cloud/storage in nodejs/typescript app

Viewed 425

I'm downloading files from GCP Cloud Storage bucket from a NodeJS/Express app with Typescript, using the official library @google-cloud/storage. I'm running application locally, inside docker image that runs on docker-compose. Standard local environment, I guess. The problem is that the file download takes so much time, and I really don't understand why is so. In fact, I tried to download files with GCP REST API (through media link url), using simple fetch request: in this case, everything goes well and the download time is ok. Here below download time comparison for a couple of files of different dimensions:

  1. 1KB: @google-cloud/storage 621 ms, fetch 224 ms
  2. 587KB: @google-cloud/storage 4.1 s, fetch 776 ms
  3. 28MB: @google-cloud/storage 2 minutes and 4 seconds, fetch 4 s

@google-cloud/storage authentication is managed through GOOGLE_APPLICATION_CREDENTIALS environment variable. I have the same problems with both 5.8.5 and 5.14.0 versions of the @google-cloud/storage library. Precisely, I need to get the file as buffer in order to directly manage its content in the Node application, below the code.

import fetch from 'node-fetch'
import { Storage as GoogleCloudStorageLibrary } from '@google-cloud/storage'

export interface GoogleCloudStorageDownload {
  fileBuffer: Buffer;
  fileName: string;
}

// this method takes long time to retrieve the file and resolve the promise
const downloadBufferFile = async (filePath: string, originalName: string): Promise<GoogleCloudStorageDownload> => {
  const storage = new GoogleCloudStorageLibrary()
  const bucket = storage.bucket('...gcp_cloud_storage_bucket_name...')

  return new Promise<GoogleCloudStorageDownload>((resolve, reject) => {
    bucket
      .file(filePath)
      .download()
      .then((data) => {
        if (Array.isArray(data) && data.length > 0 && data[0]) {
          resolve({ fileBuffer: data[0], fileName: originalName })
        }
      })
      .catch((e) => {
        if (e.code === 404) {
          reject(new Error(`CloudStorageService - ${e.message} at path: ${filePath}`))
        }

        reject(new Error(`Error in downloading file from Google Cloud bucket at path: ${filePath}`))
      })
  })
}

// this method takes normal time to retrieve the file and resolve
const downloadBufferFileFetch = async (filePath: string, originalName: string): Promise<GoogleCloudStorageDownload> {
  const fetchParams = {
    headers: {
      Authorization: 'Bearer ...oauth2_bearer_token...'
    }
  }

  const fetchResponse = await fetch(filePath, fetchParams)
  if (!fetchResponse.ok) {
    throw new Error(`Error in fetch request: ${filePath}`)
  }

  const downloadedFile = await fetchResponse.buffer()

  const result = {
    fileBuffer: downloadedFile,
    fileName: originalName
  }

  return result
}

const filePath = '...complete_file_path_at_gcp_bucket...'
const originalName = 'fileName.csv'
const slowResult = await downloadBufferFile(filePath, originalName)
const fastResult = await downloadBufferFileFetch(filePath, originalName)

The bucket has standard configuration.

You may suggest to just use the REST API with fetch but that should be not optimal and/or annoying, since I would have to manage the Authorization Bearer token and its refresh, for each environment the application will be running on.

Am I doing something wrong? What may be the cause of the very/extremely slow download?

0 Answers
Related