Memory crash on sending 100MB+ file to S3 on chrome

Question

I'm currently using Javascript to upload some video files to S3. The process works for files <100MB, but for ~100MB plus on chrome I run into an error (this works on safari). I am using ManagedUpload in this example which should be doing multipart/form-data in the background.

Code snippet:

...
        let upload = new AWS.S3.ManagedUpload({
            params:{
                    Bucket:  'my-bucket',
                    Key:   videoFileName,
                    Body:  videoHere,
                    ACL:  "public-read"
            }
        });
        upload.promise();
        
...

Chrome crashes with the error RESULT_CODE_INVALID_CMDLINE_URL, dev tools crash and in the Chrome terminal logs i get this:


[5573:0x3000000000]    27692 ms: Scavenge 567.7 (585.5) -> 567.7 (585.5) MB, 23.8 / 0.0 ms  (average mu = 0.995, current mu = 0.768) allocation failure 
[5573:0x3000000000]    28253 ms: Mark-sweep 854.6 (872.4) -> 609.4 (627.1) MB, 235.8 / 0.0 ms  (+ 2.3 ms in 2 steps since start of marking, biggest step 1.4 ms, walltime since start of marking 799 ms) (average mu = 0.940, current mu = 0.797) allocation fa

<--- JS stacktrace --->

[5573:775:0705/140126.808951:FATAL:memory.cc(38)] Out of memory. size=0
[0705/140126.813085:WARNING:process_memory_mac.cc(93)] mach_vm_read(0x7ffee4199000, 0x2000): (os/kern) invalid address (1)
[0705/140126.880084:WARNING:system_snapshot_mac.cc(42)] sysctlbyname kern.nx: No such file or directory (2)

I've tried using HTTP PUT also, both work for smaller files but once i get bigger they both crash.

Any ideas? I've been through tons of SO posts / AWS docs but nothing helped this issue yet.

Edit: I've filed the issue with Chrome; seems like its an actual bug. Will update post when I have an answer.

user_6580912 · Answer

This issue came from loading the big file into memory (several times) which would crash chrome before it even had a chance to upload.

The fix was using createObjectURL (a url pointing to the file) instead of readAsDataUrl (the entire file itself), and when sending the file to your API, use const newFile = new File([await fetch(objectURL).then(req => req.blob()], 'example.mp4', {type: 'video/mp4'});

This worked for me as I was doing many conversions to get readAsDataUrl to the file type i wanted, but in this way i use much less space.

Memory crash on sending 100MB+ file to S3 on chrome

1 Answers