In this article we will look at AWS S3 Bucket Transfer. I have a lot of data that is stored on S3. I ended up wanting to move a specific bucket from one account (in Sydney) to another account (in the US). S3 to S3 should be easy right ? Well it is ! Other reasons to do this are you could also be looking at moving the bucket to another account for cost reasons, or even just change the storage type for the bucket from Glacier to Standard.
There is already a great article from AWS that covers the steps – but a few things were not clear so Im adding in notes that I needed to do so this was a successful process.
The AWS article can be read HERE. The steps with a few more lines of detail are as follows (with a slight change in the order to flow more correctly):
STEPS
- In the DESTINATION account, you need to log in the the UI and create an IAM policy, then an IAM user with that policy attached to it. This user will then have the source and destination bucket available to control the process (but we still need to permit that from the SOURCE account)
- In the SOURCE account, create the source policy as per the doc and attach this to the SOURCE bucket that you want to copy from
- Via the AWS CLI, using the DESTINATION IAM user account you just setup, run the copy process. For my example i just setup the AWS credentials under [default] in my AWS CLI credentials/config files (see my previous post for more on these files) so I could then use the –profile option with the AWS S3 command.
That’s it ! AWS S3 Bucket Transfer done. The copy process will run and duplicate the files from the source bucket to the destination bucket. Here’s the single line command and the policies again from the AWS link (with my references to each argument). The policy files can be downloaded from the AWS link :
CODE AND CONFIG
c:\>aws s3 cp s3://[SOURCE BUCKET]/ s3://DESTINATION BUCKET/ --recursive --source-region [DESTINATION REGION] --region [DESTINATION REGION] --profile default
Source Bucket Policy
Destination IAM User Policy