How to Copy Files from Linux to S3 bucket

Amazon S3 is a low-cost, cloud-based data storage solution by Amazon Web Services. It is often used to store static web files such as CSS, JS, Images. You can easily upload files from your AWS EC2 instance to Amazon S3 bucket as well as download them from S3 bucket to your EC2 instance. In this article, we will learn how to do them both.

How to Copy Files from Linux to S3 bucket

Here are the steps to copy files from Linux to S3 bucket.

1. Install AWSCLI

AWSCLI is available in almost every default Linux repository. So you can easily install it with the following commands, depending on your Linux distribution.

$ sudo dnf install awscli    ## Fedora, Redhat and CentOS
$ sudo apt install awscli    ## Ubuntu, Debian and Linux Mint

Run the following command to verify installation.

$ aws --version

2. Copy Files from Linux to S3 Bucket

Once AWS CLI is installed on your system, you can run the following command to list the contents of specific S3 bucket.

$ aws s3 ls s3://<S3bucketName>

Here is the command to copy file from your EC2 Instance’s Linux system to Amazon S3 bucket.

$ aws s3 cp /full/path/to/file s3://<S3BucketName>

This will copy the file to the root folder of your S3 bucket.

If you want to copy it to a subfolder, say, data, you can specify it after the bucket name as shown below.

$ aws s3 cp /full/path/to/file s3://<S3BucketName>/data

3. Copy Files from Amazon S3 bucket to Linux

If you want to copy files from Amazon S3 bucket to Linux, swap the path to file and Amazon S3 bucket in the above commands.

# To copy the files from S3 to EC2
$ aws s3 cp s3://<S3BucketName>/filename /full/path/to/file

Similarly, if you want to copy file from a subfolder in your S3 bucket, mention the subfolder after S3 bucket name.

$ aws s3 cp s3://<S3BucketName>/data/filename /full/path/to/file

4. Create Shell Scripts

You can also add the above commands in a shell script to automate tasks. Here is a command to create empty shell script to upload file.

$ vi /scripts/s3_upload.sh

Add the following lines to it. Replace S3_BUCKET_NAME and BACKUP_FILENAME with the name of Amazon S3 bucket and full path of file to be uploaded respectively.

#/usr/bin/env bash 
 
S3_BUCKET_NAME=""
BACKUP_FILENAME='/home/ubuntu/data.txt'
 
${AWSCMD} cp /tmp/${BACKUP_FILENAME} s3://${S3_BUCKET_NAME}/
 
 
if [ $? -eq 0 ]; then
 echo "File successfully uploaded to s3 bucket"
else
    echo "Error in s3 upload"
fi

Save and close the file. Make it executable using the following command.

$ chmod +x ./s3_upload.sh

Run the script with following command.

$ bash ./s3_upload.sh

In this article, we have learnt how to copy files from Linux to Amazon S3 bucket, and also how to download files from S3 bucket to Linux.

How to Copy Files from Linux to S3 bucket