Control and management of data can be a strenuous task. These AWS S3 commands will help you quickly and efficiently manage your AWS S3 buckets and Data.
AWS S3 is the object storage service provided by AWS. It is the most widely used storage service from AWS that can virtually hold an infinite amount of data. It is highly available, durable, and easy to integrate with several other AWS Services.
AWS S3 can be used by people with any requirements like mobile/web application storage, big data storage, machine learning data storage, hosting static websites, and many more.
If you have been using S3 in your project, you would know that given the vast amount of storage capacity, the management of 100s of buckets and terabytes of data in these buckets can be a demanding job. We have a list of AWS S3 commands with examples that you can use to manage your AWS S3 buckets and data efficiently.
AWS CLI Setup
After you have successfully downloaded and installed the AWS CLI, you need to configure AWS Credentials to be able to access your AWS Account and services. Let us quickly run through how you can configure AWS CLI.
The first step is to create a user with programmatic access to AWS Account. Remember to check this box when you create a user for AWS CLI.
Give the permissions and create a user. At the final screen after you successfully created this user copy the Access key ID and Secret access key for this user. We will use these credentials to log in via the AWS CLI.
Now go to the terminal of your choice and run the following command.
Enter the Access key ID and Secret access key when prompted. Select any AWS region of your choice and the command output format. I personally prefer using the JSON format. This is no big deal you can always change these values later.
You can now run any AWS CLI Command in the console. Let us now go through the AWS S3 Commands.
The cp command simply copies the data to and from S3 buckets. It can be used to copy files from local to S3, from S3 to local, and between two S3 buckets. There are a lot of other parameters that you can supply with the commands.
For example, -dryrun parameter to test the command, –storage-class parameter to specify the storage class of your data in S3, other parameters to set encryption, and much more. The cp command gives you complete control over how you configure your data security in S3.
This command list all the objects in an S3 bucket. Note here that the objects with the prefix samplePrefix/ and all the sub prefixes are also displayed.
The mb command is simply used to create new S3 buckets. This is a fairly simple command but to create new buckets, the name of the new bucket should be unique across all the S3 buckets.
aws s3 mb <BUCKET_NAME>
Create a new bucket in a specific region
aws s3 mb myUniqueBucketName --region eu-west-1
The mv command simply moves the data to and from S3 buckets. Just like the cp command, mv command is used to move data from local to S3, S3 to local, or between two S3 buckets.
The only difference between the mv and the cp command is that when using the mv command the file is deleted from the source. AWS moves this file to the destination. There are a lot of options that you can specify with the command.
The sync command copies and updates files from the source to the destination just like the cp command. It is important that we understand the difference between the cp and the sync command. When you use cp it copies data from source to destination even if the data already exists in the destination.
It will also not delete files from the destination if they are deleted from the source. However, sync looks at the destination before copying your data and only copies the new and updated files. The sync command is similar to committing and pushing changes to a remote branch in git. The sync command offers a lot of options to customize the command.
aws s3 sync <SOURCE> <DESTINATION> [--options]
Sync local folder to S3
aws s3 sync ./local_folder s3://bucket_name
Sync S3 data to a local folder
aws s3 sync s3://bucket_name ./local_folder
Sync data between two S3 buckets
aws s3 sync s3://bucket_name s3://bucket_name_2
Move data between two S3 buckets excluding all .txt files
I hope the above gives you an idea about some of the frequently used AWS S3 commands to manage buckets. If you are interested in learning more, you may check out AWS certification details.
Naman Yash is a Software Engineering Professional with 2+ years of Cloud Engineering experience in JP Morgan Chase. Currently, Naman is working as a freelance software engineer and content writer. He holds multiple AWS and Terraform certifications… read more