Background on the problem
We are trying to compress a video using FFmpeg in our server that is hosted in the Google App Engine (GAE). The input file is in Google Cloud Storage (GCS) which can be easily passed as an input to FFmpeg and the processing is done, however, I wanted to output the file directly to GCS. I have referred to the documentation here https://ffmpeg.org/ffmpeg-protocols.html#http that they provided and used the correct headers and method to upload the file using a signed URL, but it doesn't seem to work. As of now, I am not even sure whether this is possible.
Current Implementation
Currently, we are first saving the output file in the GAE workspace and then uploading the file to the bucket. This flow worked fine until we faced another problem. Whenever the file is too big, the processing takes more than 10 mins which seems to be the threshold for Automatically Scaled Instances in the Flexible environment. To get around this problem we could use basic_scaling in a Standard environment but there we cannot seem to write files in the GAE workspace. I could choose to write in the /tmp directory but that uses RAM and we can have many files being uploaded at a time, so it's not an option.
Possible Future
Right now one solution I have seen is to use a Flexible environment with manual scaling, but this is a bad idea when it comes to scaling and cost-effectiveness. Another, which I am not so sure about would be to use Google Compute Engine but I am yet to try this.
Conclusion
The problem that arose from not being able to make a PUT request to Google Cloud Storage from FFmpeg led me to go through tens of problems surrounding GAE and its weird combination of instances and feature set.
Would really appreciate suggestions or possible solutions if I am missing anything. If only we could make the FFmpeg output the file to GCS.