Use Cloud Function download a JSON file from URL then upload to a Cloud Storage bucket, status 200 but JSON file uploaded is only 20 bytes and empty

I don't write much NodeJS but I think your issue is with async code.

You create the stream and then issue the http.get but you don't block on the callback (piping the file) completing before you start the GCS upload.

You may want to attach an .on("finish", () => {...}) to the pipe and in that callback, upload the file to GCS.

NOTE IIRC GCS has a method that will let you write a stream directly from memory rather than going through a file.

NOTE if you pull the storage object up into the global namespace, it will only be created whenever the instance is created and not every time the function is invoked.

As pointed by @DazWilkin, there are issues with asynchronous code. You must wait for onfinish() to trigger and then proceed. Also the upload() method returns a promise too. Try refactoring your function in async-await syntax as shown below:

exports.writeToBucket = async (req, res) => {
  const http = require('http');
  const fs = require('fs');

  // Imports the Google Cloud client library
  const {Storage} = require('@google-cloud/storage');
    
  // Creates a client
  const storage = new Storage();
  const bucketName = 'tft-test-48c87.appspot.com';
  const filename = '/tmp/post.json';  

  await downloadJson()

  // Uploads a local file to the bucket
  await storage.bucket(bucketName).upload(filename, {
    gzip: true,
    metadata: {
      cacheControl: 'no-cache',
    },
  });

  res.status(200).send(`${filename} uploaded to ${bucketName}.`);
}

const downloadJson = async () => {
 const Axios = require('axios')
 const fs = require("fs")
 const writer = fs.createWriteStream("/tmp/post.json")
 const response = await Axios({
   url: "http://jsonplaceholder.typicode.com/posts",
   method: 'GET',
   responseType: 'stream'
 })
 response.data.pipe(writer)
 return new Promise((resolve, reject) => {
   writer.on('finish', resolve)
   writer.on('error', reject)
 })
}

This example uses Axios but you can do the same with http.

Do note that you can directly upload the fetched JSON as a file like this:

exports.writeToBucket = async (req, res) => {
  const Axios = require("axios");
  const { Storage } = require("@google-cloud/storage");

  const storage = new Storage();
  const bucketName = "tft-test-48c87.appspot.com";
  const filename = "/tmp/post.json";

  const { data } = await Axios.get("http://jsonplaceholder.typicode.com/posts");

  const file = storage.bucket(bucketName).file("file.json");
  const contents = JSON.stringify(data);
  await file.save(contents);

  res.status(200).send(`${filename} uploaded to ${bucketName}.`);
};

You can read more about the save() method in the documentation.

You don't need a write stream to get the URL data, fetch the URL, await the response to resolve, call the appropriate response.toJson() method.

Personally, I prefer to use Fetch and Axios over http as they are cleaner to work with. But with Nodes http you can do the following:

https.get(url,(res) => {
    let body = "";

    res.on("data", (chunk) => {
        body += chunk;
    });

    res.on("end", () => {
        try {
            let json = JSON.parse(body);
            // do something with JSON
        } catch (error) {
            console.error(error.message);
        };
    });

}).on("error", (error) => {
    console.error(error.message);
});

Once you have that, you can pass it directly to a storage method as a data blob or byte array.

byte[] byteArray = resultJson.toString().getBytes("UTF-8");