Rails 5 - Concurrent large Video uploads and FFMPEG encoding in the background is making the server very slow

user_1482055 0
Viewed 776

I have a working Rails 5 apps using Reactjs for frontend and React dropzone uploader to upload video files using carrierwave.

So far, what is working great is listed below -

  1. User can upload videos and videos are encoded based on the selection made by user - HLS or MPEG-DASH for online streaming.
  2. Once the video is uploaded on the server, it starts streaming it by:-
    • FIRST,copying video on /tmp folder.
    • Running a bash script that uses ffmpeg to transcode uploaded video using predefined commands to produce new fragments of videos inside /tmp folder.
    • Once the background job is done, all the videos are uploaded on AWS S3, which is how the default carrierwave works
  3. So, when multiple videos are uploaded, they are all copied in /tmp folder and then transcoded and eventually uploaded to S3.

My questions, where i am looking some help are listed below -

1- The above process is good for small videos, BUT what if there are many concurrent users uploading 2GB of videos? I know this will kill my server as my /tmp folder will keep on increasing and consume all the memory, making it to die hard.How can I allow concurrent videos to upload videos without effecting my server's memory consumption?

2- Is there a way where I can directly upload the videos on AWS-S3 first, and then use one more proxy server/child application to encode videos from S3, download it to the child server, convert it and again upload it to the destination? but this is almost the same but doing it on cloud, where memory consumption can be on-demand but will be not cost-effective.

3- Is there some easy and cost-effective way by which I can upload large videos, transcode them and upload it to AWS S3, without effecting my server memory. Am i missing some technical architecture here.

4- How Youtube/Netflix works, I know they do the same thing in a smart way but can someone help me to improve this?

Thanks in advance.

carrierwave video-streaming ffmpeg amazon-s3 ruby-on-rails
4 Answers

FFmpeg has some options to connect from a stream, so you can connect to your S3 file directly and process it by small parts and upload as multipart to S3 too you can do this right with the server that you already have or use lambda from AWS as @samuil said

here is a question where they have a simple snippet that could send you in the right direction

var s3Client = new AmazonS3Client(RegionEndpoint.USEast1);

var startInfo = new ProcessStartInfo
{
    FileName = "ffmpeg",
    Arguments = $"-i pipe:0 -y -vn -ar 44100 -ab 192k -f mp3 pipe:1",
    CreateNoWindow = true,
    RedirectStandardInput = false,
    RedirectStandardOutput = false,
    UseShellExecute = false,
    RedirectStandardInput = true,
    RedirectStandardOutput = true,
};

using (var process = new Process { StartInfo = startInfo })
{
    // Get a stream to an object stored on S3.
    var s3InputObject = await s3Client.GetObjectAsync(new GetObjectRequest
    {
        BucketName = "my-bucket",
        Key = "input.wav",
    });

    process.Start();

    // Store the output of ffmpeg directly on S3 in a background thread
    // since I don't 'await'.
    var uploadTask = s3Client.PutObjectAsync(new PutObjectRequest
    {
        BucketName = "my-bucket",
        Key = "output.wav",
        InputStream = process.StandardOutput.BaseStream,
    });

    // Feed the S3 input stream into ffmpeg
    await s3Object.ResponseStream.CopyToAsync(process.StandardInput.BaseStream);
    process.StandardInput.Close();

    // Wait for FFmpeg to be done
    await uploadTask;

    process.WaitForExit();
}
user_20435920

One convenient option for doing that would be:

  1. Upload files directly to S3 using presigned URLs
  2. Use AWS Lambda for transcoding, or even better: use hosted & managed solution for that.

Concurrent users won't be your limitation: you don't really care about disk OR computational resources here. Spikey traffic will be handled by managed service. While solution running costs might be a bit higher when compared to self-built solution, I doubt it will be cheaper if you factor in infrastructure setup and maintenance costs.

user_1155760

I assume that this question considers your limited resources both on machine & money.

Consider using queue for ffmepg encoding script - then you will avoid simultaneous encoding tasks.

user_2519200

After trying bunch of options, I was able to come up with a fool-proof and scalable option, by which I am able to upload almost 10GB of video easily, The steps, in concise manner are listed below -

  1. No more server side uploading for large videos. Hence Carrierwave is removed.
  2. Instead, I employed aws-sdk library to upload directly from the Reactjs Frontend.
  3. This involves a single trip back to the server to ensure file(key) is successfully uploaded and then later updates the DB.
  4. From AWS's perspective, the chain goes like ->> S3(object created) >> Triggers Lambda Job >> Job fetches object from input bucket and after conversion dumps into output bucket >> also hits the server with Job ID to give it to the backend which is stored in the db(to let users know if video os still processed or not) >> then SNS sends an email to the user.
  5. S3 Output bucket is mounted with CloundFront along with Accleration enabled, to allow fast upload(5gb in hardly 1.10 mins).

As there are metadata details of the uploaded content, the backend uses it to serve and update the users accordingly.

All the above was totally inspired and customised based on the existing AWS Video ON demand documentation on Github. Link is here - https://github.com/aws-solutions/video-on-demand-on-aws

Hope it helps.

user_14820550
Related