What is the algorithm to compute the Amazon-S3 Etag for a file larger than 5GB

Question

Files uploaded to Amazon S3 that are smaller than 5GB have an ETag that is simply the MD5 hash of the file, which makes it easy to check if your local files are the same as what you put on S3.

But if your file is larger than 5GB, then Amazon computes the ETag differently.

For example, I did a multipart upload of a 5,970,150,664 byte file in 380 parts. Now S3 shows it to have an ETag of 6bcf86bed8807b8e78f0fc6e0a53079d-380. My local file has an md5 hash of 702242d3703818ddefe6bf7da2bed757. I think the number after the dash is the number of parts in the multipart upload.

I also suspect that the new ETag is still an MD5 hash, but with some metadata included along the way from the multipart upload somehow.

Does anyone know how to compute the ETag using the same algorithm as Amazon S3? Any help will be appreciated!!

Answer 1

Say you uploaded a 14MB file to a bucket without server-side encryption, and your part size is 5MB. Calculate 3 MD5 checksums corresponding to each part, i.e. the checksum of the first 5MB, the second 5MB, and the last 4MB. Then take the checksum of their concatenation. MD5 checksums are often printed as hex representations of binary data, so make sure you take the MD5 of the decoded binary concatenation, not of the ASCII or UTF-8 encoded concatenation. Add a hyphen and the number of parts to get the ETag when that's done.

Here are the commands to do it on Mac OS X from the console:

$ dd bs=1m count=5 skip=0 if=someFile | md5 >>checksums.txt 5+0 records in 5+0 records out 5242880 bytes transferred in 0.019611 secs (267345449 bytes/sec) $ dd bs=1m count=5 skip=5 if=someFile | md5 >>checksums.txt 5+0 records in 5+0 records out 5242880 bytes transferred in 0.019182 secs (273323380 bytes/sec) $ dd bs=1m count=5 skip=10 if=someFile | md5 >>checksums.txt 2+1 records in 2+1 records out 2599812 bytes transferred in 0.011112 secs (233964895 bytes/sec)

At this point, all the checksums are in checksums.txt. To concatenate them and decode the hex and get the MD5 checksum of the lot, just use

$ xxd -r -p checksums.txt | md5

Now append "-3" to get the ETag, since there were 3 parts. Hope this answers your doubt!!