Java hanging on FileOutputStream.close() for S3-to-EC2 Download

I have an application that's hanging on FileOutputStream.close() (well, really it's hanging on the native FileOutputStream.close0() but whatever). And... I don't really know what would cause that. How would an expert at Java/Linux debugging approach this problem?

I've used jstack to find the crime scene, and here it is:

s3-transfer-manager-worker-1" #22 prio=5 os_prio=0 tid=0x00007f9f40007800 nid=0x2e6 runnable [0x00007f9fac4d7000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileOutputStream.close0(Native Method)
        at java.io.FileOutputStream.access$000(FileOutputStream.java:53)
        at java.io.FileOutputStream$1.close(FileOutputStream.java:356)
        at java.io.FileDescriptor.closeAll(FileDescriptor.java:212)
        - locked <0x00000005c3bbc440> (a java.io.FileDescriptor)
        at java.io.FileOutputStream.close(FileOutputStream.java:354)
        at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:139)
        at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
        - locked <0x00000005c3bbc600> (a java.lang.Object)
        at com.amazonaws.util.IOUtils.closeQuietly(IOUtils.java:70)
        at com.amazonaws.services.s3.internal.ServiceUtils.appendFile(ServiceUtils.java:473)
        at com.amazonaws.services.s3.transfer.internal.CompleteMultipartDownload.call(CompleteMultipartDownload.java:46)
        at com.amazonaws.services.s3.transfer.internal.CompleteMultipartDownload.call(CompleteMultipartDownload.java:29)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Download Code

Here's a simplified version of our S3 download code:

AmazonS3Client s3Client = new AmazonS3Client(new DefaultAWSCredentialsProviderChain());
s3Client.setRegion(...);
TransferManager s3TransferManager = TransferManagerBuilder.standard().withS3Client(s3Client).build();

GetObjectRequest objectRequest = new GetObjectRequest(...);
Path localTempFile = Files.createTempFile(...);
Download downloadHandle = s33TransferManager.download(objectRequest, localTempFile.toFile());

downloadHandle.waitForCompletion();

Pretty vanilla.

Background

This is a pretty vanilla usage of the Java AWS S3 API to download a file to the local disk. I can download the same file using the aws s3 cp ... command line application just fine. Even more confusingly, the application was working just fine previously. It has about 1000 of these files to download, and it made it through 11 of them before this started happening.

Things I know:

• These are beefy files, about 20 GB each.
• I've tried multiple times, and let it sit there for 1:00h and lately, for 2:00h. It stays hung.
• We have plenty of free disk space: 200 GB. (And it'd be throwing an exception if that were the problem, anyways.)
• I tried turning it off and then on again (via sudo reboot).
• It's an m4.2xlarge EC2 instance.
• Per CloudWatch, the CPU isn't busy.
• I can download the exact same file from S3 to the exact same location via the aws s3 cp ... command line, which rules out:
  • Disk space problems (again).
  • IOPS shortages (unless the Java SDK's TransferManager is doing something really different and dumb).
  • Really slow downloads: it only takes about 5 minutes to download the file on the command line.
  • File permission problems.

Next Steps?

Unfortunately, I don't know enough to go any further than that in debugging this. Are there specific Linux metrics or commands I should be checking to see if some system resource has been exhausted or deadlocked?