The AWS CLI makes working with files in S3 very easy. However, the file globbing available on most Unix/Linux systems is not quite as easy to use with the AWS CLI. S3 doesn’t have folders, but it does use the concept of folders by using the “/” character in S3 object keys as a folder delimiter.
To copy all objects in an S3 bucket to your local machine simply use the
aws s3 cp command with the
aws s3 cp s3://temp-bucket/ ./ --recursive will copy all files from the “big-datums-tmp” bucket to the current working directory on your local machine. If there are folders represented in the object keys (keys containing “/” characters), they will be downloaded as separate directories in the target location.
aws s3 cp s3://temp-bucket/folder1/ ./ --recursive is almost the same as the one above, but this command will only copy files from “myFolder” folder (objects with keys starting with “myFolder/”).
For using wildcards and patterns to copy only certain files, refer to Using Wildcards with AWS CLI on how to correctly use the