I am attempting to upload large files (20GB+) to Amazon S3.
My server component generates pre-signed URLs so that I do not need to send traffic via my application server. Instead, I do a PUT request directly to S3.
open System
open System.IO
open System.Net.Http
open System.Security.Cryptography
open System.Text
// ...
task {
use fileStream = File.OpenRead(filePath)
use putRequest = new HttpRequestMessage()
putRequest.RequestUri <- Uri preSignedUrl
putRequest.Method <- HttpMethod.Put
putRequest.Content <- new StreamContent(fileStream)
let! response = context.SendRequest(putRequest)
let _ = response.EnsureSuccessStatusCode()
printfn "Done. "
}
However, for large files (limit seems to be around 10GB) I hit this error:
System.AggregateException: One or more errors occurred. (Error while copying content to a stream.)
---> System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe.
---> System.Net.Sockets.SocketException (32): Broken pipe
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken)
at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Security.SslStream.WriteAsyncChunked[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory`1 buffer)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory`1 source, Boolean async)
at System.Net.Http.HttpConnection.WriteAsync(ReadOnlyMemory`1 source, Boolean async)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnection.ContentLengthWriteStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken ignored)
at System.IO.Stream.<CopyToAsync>g__Core|29_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at Microsoft.Win32.SafeHandles.SafeFileHandle.ThreadPoolValueTaskSource.ExecuteInternal()
at Microsoft.Win32.SafeHandles.SafeFileHandle.ThreadPoolValueTaskSource.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
--- End of stack trace from previous location ---
How can I upload large files to S3?
Not I cannot use the aws CLI since the client is not fully trusted.