Does AWS CLI upload files larger than 4GB to Amazon Glacier?

user_1032490 0
Viewed 2003

The AWS Command Line Interface (CLI) allows to upload a file to AWS Glacier. But there is also a limit of 4GB for file uploads in the AWS Rest API. If I need to upload a file larger than 4GB through the Rest API, I need to use the multi-part upload.

My question is: does the AWS CLI handle internally file uploads larger than 4GB, or do I need to handle myself the multipart upload when handling files larger than 4GB? Can I just pass a 20Gb file to the upload-archive option of the AWS CLI and it will just work? If the CLI can't handle large file uploads directly, there is any command line tool that does it for me (freeing me from the trouble of implementing all of the checksum computing, error handling and retry logic when a part upload fails)?

I understand that the 4GB limit is on the AWS Rest API, but I could not find anything about how this limit is handled in the CLI. I could just make the test, but my upload speed is not so fast and I fear wasting a few hours before discovering that it does not work.

amazon-glacier aws-cli amazon-web-services
2 Answers

The below script will work fine. I had created chunks for treehash calculation and fileparts uploads saperately. It worked fine. 

#!/bin/bash

date1=$(date +"%s")

byteSize=1073741824

CHUNK_SIZE=1073741824

hashsize=1048576

if [[ -z "${1}" ]]; then
    echo "No file provided."
    exit 1
fi

ARCHIVE="/mnt/dbfiles/mahipal/splitfiles/${1}"
ARCHIVE_SIZE=`cat "${ARCHIVE}" | wc --bytes`

cd /mnt/dbfiles/mahipal/splitfiles

rm -rf TEMP

rm -rf HASH

mkdir TEMP

mkdir HASH

cd /mnt/dbfiles/mahipal/splitfiles/TEMP

date3=$(date +"%s")

split -d --bytes=${CHUNK_SIZE} "${ARCHIVE}" chunk  -a 4

date4=$(date +"%s")

diff2=$(($date4-$date3))

cd /mnt/dbfiles/mahipal/splitfiles/HASH 

date5=$(date +"%s")

split -d --bytes=${hashsize} "${ARCHIVE}" chunk  -a 5

date6=$(date +"%s")

diff3=$(($date6-$date5))

cd /mnt/dbfiles/mahipal/splitfiles/TEMP

lastpartsize=`expr $(ls -l | tail -1 | awk '{print$5}') + 0`

lastfile=$(ls -l | tail -1 | awk '{print$9}')

cont=$(ls -l | wc -l)
cnt=`expr $cont - 2`

fileCount=$(ls -1 | grep "^chunk" | wc -l)
echo "Total parts to upload: " $fileCount

files=$(ls | grep "^chunk")

init=$(/bin/aws glacier initiate-multipart-upload --account-id - --part-size $byteSize --vault-name final_vault --archive-description "${1}_${ARCHIVE_SIZE}_${byteSize}")

echo "---------------------------------------"


uploadId=$(echo $init | jq '.uploadId' | xargs)

touch commands.txt

i=0
for f in $files 
  do
     byteStart=$((i*byteSize))
     byteEnd=$((i*byteSize+byteSize-1))
     echo /bin/aws glacier upload-multipart-part --body $f --range "'"'bytes '"$byteStart"'-'"$byteEnd"'/*'"'" --account-id - --vault-name final_vault --upload-id $uploadId >> commands.txt
     i=$(($i+1))

    if [ "$i" == "$cnt" ]
        then
    byteEnd=`expr $byteEnd + 1`
    byteEnd2=$((i*byteSize+lastpartsize-1))
    byteSize=$lastpartsize 
     echo /bin/aws glacier upload-multipart-part --body $lastfile --range "'"'bytes '"$byteEnd"'-'"$byteEnd2"'/*'"'" --account-id - --vault-name final_vault --upload-id $uploadId >> commands.txt
    break   
    fi      

  done

parallel --load 100% -a commands.txt --no-notice --bar

cd /mnt/dbfiles/mahipal/splitfiles/HASH

files=$(ls | grep "^chunk")

for f in $files
  do
     openssl dgst -sha256 -binary ${f} > "hash${f:5}"

  done


echo "List Active Multipart Uploads:"
echo "Verify that a connection is open:"

/bin/aws glacier list-multipart-uploads --account-id - --vault-name final_vault >> /mnt/dbfiles/mahipal/splitfiles/TEMP/commands.txt

echo "-------------"
echo "Contents of commands.txt"

cd /mnt/dbfiles/mahipal/splitfiles/TEMP

cat commands.txt

# Calculate tree hash.

cd /mnt/dbfiles/mahipal/splitfiles/HASH

echo "Calculating tree hash..."
while true; do
    COUNT=`ls hash* | wc -l`
    if [[ ${COUNT} -le 2 ]]; then
        TREE_HASH=$(cat hash* | openssl dgst -sha256 | awk '{print $2}')
        break
    fi
    ls hash* | xargs -n 2 | while read PAIR; do
        PAIRARRAY=(${PAIR})
        if [[ ${#PAIRARRAY[@]} -eq 1 ]]; then
            break
        fi
        cat ${PAIR} | openssl dgst -sha256 -binary > temphash
        rm ${PAIR}
        mv temphash "${PAIRARRAY[0]}"
    done
done

cd /mnt/dbfiles/mahipal/splitfiles/TEMP

echo "Finalizing..."
/bin/aws glacier complete-multipart-upload  --account-id=- --vault-name="final_vault" --upload-id="$uploadId" --checksum="${TREE_HASH}" --archive-size=${ARCHIVE_SIZE} >>commands.txt
RETVAL=$?
if [[ ${RETVAL} -ne 0 ]]; then
    echo "complete-multipart-upload failed with status code: ${RETVAL}" >>commands.txt
    echo "Aborting upload ${uploadId}"  >>commands.txt
    /bin/aws glacier abort-multipart-upload --account-id=- --vault-name="final_vault" --upload-id="${uploadId}" >>commands.txt
    exit 1
fi

echo "--------------"
echo "Deleting temporary commands.txt file"

#rm commands.txt

date2=$(date +"%s")

diff=$(($date2-$date1))

echo "Total Split Duration for Chunk Part Size: $(($diff2/ 3600 )) hours $((($diff2 % 3600) / 60)) minutes $(($diff2 % 60)) seconds" >>commands.txt 

echo "Total Split Duration for hash Part Size: $(($diff3/ 3600 )) hours $((($diff3 % 3600) / 60)) minutes $(($diff3 % 60)) seconds"  >>commands.txt

echo "Total upload Duration: $(($diff/ 3600 )) hours $((($diff % 3600) / 60)) minutes $(($diff % 60)) seconds" >>commands.txt

echo "Done."
exit 0
user_97942570
Related