how to createWriteStream() to GCS?

Question

I'm trying to write an Express route that takes an image URI in the POST body and then saves the image into a Google Cloud Storage Bucket.

I'm not able to persist this image to local disk and need to stream the buffer straight to the GCS bucket.

My route creates a 4KB "stub" in the GCS bucket but there's no image payload. My nodejs then proceeds to crash...

Q: What is the correct way to .pipe() the results of the https.request() to blob.createWriteStream()? Is this the right approach?

I've spent a few days trying to find the right solution with different stream handlers but with precious little progress to show for it. Can someone help?

message: 'The rate of change requests to the object my-projectID/testimage.jpg exceeds the rate limit. Please reduce the rate of create, update, and delete requests.'

  const streamifier = require('streamifier');
  const {Storage} = require('@google-cloud/storage');
  const storage = new Storage({
      projectId: 'my-projectID',
      keyFile: '../config/my-projectID.json'
      });
  const bucket = storage.bucket('my-projectID');
  const blob = bucket.file('testimg.jpg');

app.post('/bam', passport.authenticate('basic', {session: false }), (req, res) => {
    
return new Promise((resolve, reject) => {
        https.request(req.body.pic, (response) => {            
            response.on('data', (d) => {
                streamifier.createReadStream(d)
                .pipe(blob.createWriteStream({
                    resumable:false, 
                    public:true,
                    metadata:{ contentType: 'image/jpeg' }
                    }));
            });
            
            response.on('finish', (done) => {
                console.log(done);
            });

            response.on('error', (err) => {
                console.log(err);
            });

        }).end();
    }); 

});

** Apologies for my ugly JS, I'm still at the bottom of ES6 learning curve.

user_10225930 · Answer

I haven't tried with https.request(), but I was able to upload an image directly to GCS without storing it locally by following the Cloud Storage Node.js Client:

const fs = require('fs')
const request = require('request')
const {Storage} = require('@google-cloud/storage');
const storage = new Storage();
const myBucket = storage.bucket('my-bucket');
const blob = myBucket.file('image.jpg');
 
const uploadblob = (url, callback) => {
  request.head(url, (err, res, body) => {
    request(url)
      .pipe(blob.createWriteStream())
      .on('close', callback)
  })
}
const url = 'https://www.example/image.jpg'
 
uploadblob(url, () => {console.log('Done!')})

user_9524453 · Answer

If your goal is to save an image to GCS you have to create a Promise and then resolve it. According to the Google Cloud Storage documentation you have to instantiate a new Storage object, then pointing at the bucket you want, then create a promise that upload the file to the bucket via "createWriteStream" and then resolve the promise you want. I.E. here I'm uploading a file to the bucket and then returning the public url. If you take a look to this code, this is the right way to create a WriteStream to GCS. It's a little bit more complex and different function than yours just cause here you can upload multiple files with a foreach loop but the process to create a stream to GCS basically remains the same. This is my function in the controller:

controllers/postControllers.js:

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();
const bucket = storage.bucket(process.env.GCLOUD_STORAGE_BUCKET);

//Func to upload files to GCS
const uploadFileTGcs = async (file) => {
  let promises = [];
  _.forEach(file, (value, key) => {
    const {originalname, buffer} = value;
    const blob = bucket.file(originalname.replace(/ /g, '_'));

    const promise = new Promise((resolve, reject) => {
      const blobStream = blob.createWriteStream({
        resumable: false,
        public: true,
      });
      blobStream.on('error', () => {
        reject(`Unable to upload image, something went wrong`);
      }).on('finish', async () => {
        const publicUrl = `https://storage.googleapis.com/${bucket.name}/${blob.name}`;
        resolve(publicUrl);
      }).end(buffer);
    });
    promises.push(promise);
  });
  return Promise.all(promises).then(promises => {
    return promises;
  });
};

Then in the route I can use that function as a middleware:

router.post('/create', async (req, res, next) => {
  try {
    if (!req.files) {
      res.status(400).json({
        messages: 'No file uploaded',
      });
      return;
    }

    const promiseReturned = await postsController.uploadFileTGcs(req.files);

    ....

    res.status(promiseReturned ? 200 : 404).json({
      result: promiseReturned, //here I'm returning the url of the files stored in gcs
      message: 'Post created',
    });
  } catch (e) {
    res.status(500).json({
      result: e.toString(),
    });
  }
});

how to createWriteStream() to GCS?

2 Answers