Ubuntu 22.04 Backup using Amazon S3 and rclone

Home > Ubuntu > Ubuntu 22.04 > Ubuntu 22.04 Backup using Amazon S3 and rclone

Creating 7 Daily Buckets with 6-Day Immutability

• Log in to your AWS account and open the S3 console.

• Click Create Bucket, enter a unique name, and confirm creation.

• Open the newly created bucket and go to the Properties tab.

• Under Bucket Versioning, click Enable (required for object lock).

• Scroll to Object Lock, click Edit, and enable it.

• Enable Default Retention:

• Choose Compliance mode.

• Set retention period to 6 days.

• Save changes.

Writing Lifecycle Policies for Permanent Deletion after Rclone Deletes

When using rclone to delete objects from an S3 bucket, AWS retains these objects as "deleted objects" if Versioning is enabled. To permanently remove them after 1 day, a Lifecycle Policy must be created.

• Open the AWS S3 Console and your target bucket.

• Go to the Management tab > Lifecycle Rules > Create Lifecycle Rule.

• Set a name and choose to apply the rule to all objects.

• Select the following actions Under Lifecycle Rule Actions:

• Permanently delete noncurrent versions of objects

• Delete expired object delete markers or incomplete multipart uploads

• Set "Days after object deletion" to 1.

• Click Create Rule.

Creating an IAM User with Access to a Single S3 Bucket

Create the User

• Go to IAM > Users > Add User.

• Provide a username and select AWS Management Console access (for GUI access).

• Create Custom password

• uncheck the option: Users must create a new password at sign-in - Recommended

• Continue to permissions setup.

Attach a Custom Policy

• Select Attach policies directly > Create Policy.

• Switch to the JSON tab and paste the following:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"s3:GetBucketLocation",
				"s3:ListAllMyBuckets"
			],
			"Resource": "arn:aws:s3:::*"
		},
		{
			"Effect": "Allow",
			"Action": "s3:*",
			"Resource": [
				"arn:aws:s3:::<Bucket_name>",
				"arn:aws:s3:::<Bucket_name>/*"
			]
		}
	]
}

• Replace <Bucket_name> with your actual bucket name.

• Give the policy name

• This policy allows full access to the specified bucket, but not others.

Complete User Creation

• Name and create the policy.

• Return to the IAM user setup screen.

• Refresh and attach the new policy.

• Click Next and then Create User.

Note: The user will see names of other buckets but cannot access their data or delete them.

Install rclone

Fetch and unpack

# apt intall curl -y
# sudo -v ; curl https://rclone.org/install.sh | sudo bash
# curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
# unzip rclone-current-linux-amd64.zip
# cd rclone-*-linux-amd64

Copy binary file

# sudo cp rclone /usr/bin/
# sudo chown root:root /usr/bin/rclone
# sudo chmod 755 /usr/bin/rclone

Install manpage

# sudo mkdir -p /usr/local/share/man/man1
# sudo cp rclone.1 /usr/local/share/man/man1/
# sudo mandb

Create Access Key for IAM user

• Login to AWS console as root

• Go to IAM > Users

• Select the user

• Under Summary Select create access key

• Select Command Line Interface (CLI)

• Give description for the key

• click Create access key

• Save the Access key and Secret access key

NOTE: If you lose or forget your secret access key, you cannot retrieve it

• Click Done

Configure rclone

Setup rclone

# rclone config

Make new remote

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Give a name

Enter name for new remote.
name> S3

Choose S3

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Outscale, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, Selectel, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others
Storage> 4

Choose the S3 provider

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Amazon Web Services (AWS) S3
   \ (AWS)
provider> 1

select the option to enter the credentials in the next step

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 1

Give the access key

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> <access_key>

Give the Secret access key

Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> <secret_access_key>

Select the region

Option region.
Region to connect to.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Asia Pacific (Mumbai).
17 | Needs location constraint ap-south-1.
   \ (ap-south-1)
region> 17

Give the Endpoint option

Option endpoint.
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
Enter a value. Press Enter to leave empty.
endpoint> 

Select the Location constraint

Option location_constraint.
Location constraint - must be set to match the Region.
Used when creating buckets only.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
17 / Asia Pacific (Mumbai) Region
   \ (ap-south-1)
location_constraint> 17

Select Access control options

Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
acl> 1

Select server side encryption options

Option server_side_encryption.
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / None
   \ ()
 2 / AES256
   \ (AES256)
 3 / aws:kms
   \ (aws:kms)
server_side_encryption> 1

Select sse kms key id options

Option sse_kms_key_id.
If using KMS ID you must provide the ARN of Key.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / None
   \ ()
 2 / arn:aws:kms:*
   \ (arn:aws:kms:us-east-1:*)
sse_kms_key_id> 1 

Select object storage class options

Option storage_class.
The storage class to use when storing new objects in S3.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Default
   \ ()
storage_class> 1

Select advance configuration options

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Complete configuration and select the remote options

Configuration complete.
Options:
- type: s3
- provider: AWS
- access_key_id: <access key>
- secret_access_key: <secret access key>
- region: ap-south-1
- location_constraint: ap-south-1
- acl: private
Keep this "S3" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

View and exit the configuration

Current remotes:

Name                 Type
====                 ====
S3                   s3         

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

rclone commands

• copy the data to S3 bucket

# rclone copy <source-file-path> <rclone-remote-name>:<bucket-name>/<folder-name-in-bucket> --progress 

--progress: displays the progress of a task. It shows the percentage completed, estimated time, and transfer rate

• sync the data to S3 bucket

# rclone sync <source-file-path> <rclone-remote-name>:<bucket-name>/<folder-name-in-bucket>

• automate the process by including in the crontab

30 23 * * 1 /usr/bin/rclone sync <file-path> <rclone-remote-name>:<bucket-name>/<folder-name-in-bucket>

Reference:

• https://rclone.org/install/

Viewing Object Count and Bucket Size in S3

• Log in to your AWS account and go to S3.

• Open the target bucket and click the Properties tab.

• Scroll to Bucket Metrics.

• View:

• Total number of objects

• Total bucket size