I'm downloading files from GCP Cloud Storage bucket from a NodeJS/Express app with Typescript, using the official library @google-cloud/storage. I'm running application locally, inside docker image that runs on docker-compose. Standard local environment, I guess.
The problem is that the file download takes so much time, and I really don't understand why is so.
In fact, I tried to download files with GCP REST API (through media link url), using simple fetch request: in this case, everything goes well and the download time is ok.
Here below download time comparison for a couple of files of different dimensions:
- 1KB:
@google-cloud/storage621 ms,fetch224 ms - 587KB:
@google-cloud/storage4.1 s,fetch776 ms - 28MB:
@google-cloud/storage2 minutes and 4 seconds,fetch4 s
@google-cloud/storage authentication is managed through GOOGLE_APPLICATION_CREDENTIALS environment variable. I have the same problems with both 5.8.5 and 5.14.0 versions of the @google-cloud/storage library.
Precisely, I need to get the file as buffer in order to directly manage its content in the Node application, below the code.
import fetch from 'node-fetch'
import { Storage as GoogleCloudStorageLibrary } from '@google-cloud/storage'
export interface GoogleCloudStorageDownload {
fileBuffer: Buffer;
fileName: string;
}
// this method takes long time to retrieve the file and resolve the promise
const downloadBufferFile = async (filePath: string, originalName: string): Promise<GoogleCloudStorageDownload> => {
const storage = new GoogleCloudStorageLibrary()
const bucket = storage.bucket('...gcp_cloud_storage_bucket_name...')
return new Promise<GoogleCloudStorageDownload>((resolve, reject) => {
bucket
.file(filePath)
.download()
.then((data) => {
if (Array.isArray(data) && data.length > 0 && data[0]) {
resolve({ fileBuffer: data[0], fileName: originalName })
}
})
.catch((e) => {
if (e.code === 404) {
reject(new Error(`CloudStorageService - ${e.message} at path: ${filePath}`))
}
reject(new Error(`Error in downloading file from Google Cloud bucket at path: ${filePath}`))
})
})
}
// this method takes normal time to retrieve the file and resolve
const downloadBufferFileFetch = async (filePath: string, originalName: string): Promise<GoogleCloudStorageDownload> {
const fetchParams = {
headers: {
Authorization: 'Bearer ...oauth2_bearer_token...'
}
}
const fetchResponse = await fetch(filePath, fetchParams)
if (!fetchResponse.ok) {
throw new Error(`Error in fetch request: ${filePath}`)
}
const downloadedFile = await fetchResponse.buffer()
const result = {
fileBuffer: downloadedFile,
fileName: originalName
}
return result
}
const filePath = '...complete_file_path_at_gcp_bucket...'
const originalName = 'fileName.csv'
const slowResult = await downloadBufferFile(filePath, originalName)
const fastResult = await downloadBufferFileFetch(filePath, originalName)
The bucket has standard configuration.
You may suggest to just use the REST API with fetch but that should be not optimal and/or annoying, since I would have to manage the Authorization Bearer token and its refresh, for each environment the application will be running on.
Am I doing something wrong? What may be the cause of the very/extremely slow download?