S3 Backup Manager

Introduction
How It Works
Installation
Configuration
Automate with Crontab
Contribution

Introduction

S3 Backup Manager is a command-line tool built using Bash. Its primary aim is to simplify the process of managing, backing up, restoring, and viewing files from an AWS S3 bucket. With the rise of data-driven applications and the need for data integrity and recovery, this tool becomes an indispensable utility for any system administrator or developer interacting with S3 buckets.

How It Works

This tool operates with a source bucket and a backup bucket:

Source Bucket: This is where your active data resides. Any files or directories in this bucket are what you would consider "live" or "production" data.
Backup Bucket: This bucket is the destination for all backup operations. The tool uses AWS's sync command to create backups into folders named after the current date. For example, a backup performed on the 15th of August, 2023 would reside in a folder named 2023-08-15.

Example Backup Bucket Structure:
```
├── 2023-08-12
│   ├── file1.txt
│   ├── file2.txt
│   └── data/
├── 2023-08-13
│   ├── file1.txt
│   ├── file2.txt
│   └── data/
├── 2023-08-14
│   ├── file1.txt
│   ├── file2.txt
│   └── data/
```
Backup: Creates a snapshot of the source bucket and stores it into the backup bucket under a folder named with the current date.
Restore: Lets you select a specific date from the available backups and syncs its content to the source bucket, effectively restoring your data to that point in time.
Show: There are two show options:
- Show all files in the main source bucket.
- Display all files of a selected backup.

Installation

Before you begin, ensure you have the following installed:

AWS CLI: Install AWS CLI. Depending on your OS, the installation method may vary. Here's how you can install it on Ubuntu:
```
sudo apt-get install awscli
```
on the centos
```
sudo pip install awscli --upgrade
```

Please note that you should have the last version of AWS-CLI so please check official-aws-installation

JQ: This is a lightweight and flexible command-line JSON processor.
```
sudo apt-get install jq
```
on the centos
```
sudo yum install jq
```
Backup Manager Script: Ensure the backup_manager.sh script is present on your system and is executable. You can clone it from the repository and make it executable:
```
git clone [email protected]:hamid/s3-backup-manager.git
chmod +x backup_manager.sh
```

Configuration

AWS CLI Configuration: Before you can use the tool, you must configure the AWS CLI with your credentials.
```
aws configure
```
Follow the prompts to input your AWS Access Key ID, Secret Access Key, default region, and default output format.
Backup Configuration: To keep your configuration tidy and separate from the main script, all configurations are stored in backup_config.conf. Open this file and replace the placeholders with the appropriate values for your setup. The script will source this file to get the configuration values.

# Define our buckets and folder
SOURCE_BUCKET="your-source-bucket-name"
BACKUP_BUCKET="your-backup-bucket-name"
BACKUP_FOLDER="file-backups"
CURRENT_DATE=$(date +\%Y-\%m-\%d)

Automate with Crontab

To automate the backup process, you can leverage crontab. Here's an example of how you can set up the script to run automatically:

Open the crontab for editing:
```
crontab -e
```
Add the following line to run the backup at 10:10 AM every day and then exit the script:
```
10 10 * * * (echo "3"; sleep 2; echo "7") | /path/to/backup_manager.sh
```
Remember to adjust the path and time according to your preferences.

Contribution

If you think of a feature that would enhance this tool or encounter a bug, please raise an issue or submit a pull request. Your contributions are highly welcomed!

hamid / s3-backup-manager Goto Github PK