Linux 解决方案1 --- 数据同步

发布时间 2023-03-22 21:13:19作者: 白马黑衣

一、概要

1. 承上启下

(1) Rock Linux 系列4 --- SSH

(2) Rocky Linux 系列5 --- rsync

(3) Rocky Linux 系列6 --- inotify-tools

2. 问题的提出

(1) 我们需要解决什么问题?

a. 网站发布之后多个节点的文件同步;

b. 数据库备份文件发生变更后传递到备份服务器;

(2) 如何解决?

利用上文讲到的SSH和rsync和inotify,我们可以实现当某一目录(如网站或数据备份目录)发生变化时,由inotify触发rsync将已经改变的文件同步至目标服务器。

a. 首先需要搭建通过公钥验证的SSH登录方式;

b. 安装rsync和inotify;

c. 创建配置文件,内容为需要同步(监控)的目录或文件,当文件或目录变更后需要同步的目标服务器以及目标目录也需要考虑;

d. 编写脚本读取d中的配置文件并对其内容使用inotifywait进行监控,当有监控时间产生时调用rsync进行后续处理;

e. 创建Linux Service将脚本以守护进程的方式进行维护;

二、解决方案

假设我有三台服务器,192.168.3.1(master),192.168.3.2(备份1),192.168.3.3(备份2)

1. 初始化

(1) 目录

sudo mkdir -p /opt/LSync

(2) 脚本

sudo touch /opt/LSync/main.sh
sudo chmod +x /opt/LSync/main.sh

(3) 配置文件config_watch_directories

sudo touch /opt/LSync/config_watch_directories

内容:

/tmp/backup/dir1
/tmp/backup/dir2

(4) 配置文件config_sync_servers

sudo touch /opt/LSync/config_sync_servers

内容:

bkuser@192.168.3.2:/home/bkuser/backup
bkuser@192.168.3.3:/home/bkuser/backup

(5) 日志

sudo touch /opt/LSync/lsync.log

(6) 权限

sudo chmod -R 755 /opt/LSync

2. SSH

(1) 登录192.168.3.1

ssh-keygen -t rsa -b 4096
chmod 700 ~/.ssh
ssh-copy-id bkuser@192.168.3.2
ssh-copy-id bkuser@192.168.3.3

(2) 登录192.168.3.2

sudo vi /etc/ssh/sshd_config
PubkeyAuthentication yes --确保开启
sudo systemctl restart sshd

(3) 登录192.168.3.3

sudo vi /etc/ssh/sshd_config
PubkeyAuthentication yes --确保开启
sudo systemctl restart sshd

3. 脚本

(1) 创建

sudo touch /opt/LSync/main.sh
sudo chmod +x /opt/LSync/main.sh

(2) 初始化

#!/bin/bash
export LC_ALL=C

DIR_BASE="/opt/LSync"

FILE_NAME_DATE="$(date +%Y%m%d)"
TIME="$(date +'%Y-%m-%d %H:%M:%S')"

PATH_CONFIG_4_WATCH="${DIR_BASE}/config_watch_directories"
PATH_CONFIG_4_SERVER="${DIR_BASE}/config_sync_servers"

DIR_LOG="${DIR_BASE}/logs"
PATH_LOG="${DIR_LOG}/${FILE_NAME_DATE}.log"

error() {
    printf "[${TIME}]-[%s]: %s\n" "$(basename "${BASH_SOURCE}")" "${1}" >&2
    exit 1
}

info() {
    printf "[${TIME}]-[%s]: %s\n" "$(basename "${BASH_SOURCE}")" "${1}" >&1
}

trap 'error "An unexpected error occurred."' ERR

init_log() {
    if [ ! -d "${DIR_LOG}" ]; then
        mkdir -p ${DIR_LOG}

        if [ "$?" -ne "0" ]; then
            error "Create ${DIR_LOG} failed."
        fi
    fi

    if [ ! -r "${PATH_LOG}" ]; then
        touch $PATH_LOG
    fi
}

verify() {
    # Check whether the config file is available
    if [ ! -r "${PATH_CONFIG_4_WATCH}" ]; then
        error "Lack of config file: ${PATH_CONFIG_4_WATCH}"
    fi

    # Check whether the config file is empty
    if [ ! -s "${PATH_CONFIG_4_WATCH}" ]; then
        error "Empty config file: ${PATH_CONFIG_4_WATCH}"
    fi

    # Check whether the config file is available
    if [ ! -r "${PATH_CONFIG_4_SERVER}" ]; then
        error "Lack of config file: ${PATH_CONFIG_4_SERVER}"
    fi

    # Check whether the config file is empty
    if [ ! -s "${PATH_CONFIG_4_SERVER}" ]; then
        error "Empty config file: ${PATH_CONFIG_4_SERVER}"
    fi
}

do_sync() {
    info "Start sync ${1} to ${2}"
    rsync -ahrvz --delete -e "ssh -i /home/[User]/.ssh/id_rsa" $1 $2
}

do_transfer() {
    source_path=$1

    while IFS= read -r line; do
        do_sync $source_path $line
    done <$PATH_CONFIG_4_SERVER
}

init_transfer() {
    while IFS= read -r line; do
        do_transfer $line
    done <$PATH_CONFIG_4_WATCH
}

init() {
    init_log
    init_transfer
}

main() {
    verify

    inotifywait -m -r -e modify,create,delete --excludei "\.(swp|swx)" --fromfile "${PATH_CONFIG_4_WATCH}" | while read path events file; do
        info "Events: ${events}, path: ${path}, file: ${file}"
        init_log

        while IFS= read -r line; do
            if [ "$line" = "${path:0:${#line}}" ]; then
                do_transfer $line
                break
            fi
        done <$PATH_CONFIG_4_WATCH
    done
}

init && main | tee -a ${PATH_LOG}

4. Service

(1) 创建服务文件

sudo touch /etc/systemd/system/lsync.service
sudo chmod 755 /etc/systemd/system/lsync.service

(2) 初始化

[Unit]
Description=Linux file synchronization system
After=network.target

[Service]
User=root
Group=root ExecStart
=/opt/LSync/main.sh [Install] WantedBy=multi-user.target

(3) 启动

sudo systemctl daemon-reload
sudo
systemctl enable --now lsync.service systemctl status lsync.service

三、参考

https://linuxhint.com/inotofy-rsync-bash-live-backups/

https://bartsimons.me/sync-folders-and-files-on-linux-with-rsync-and-inotify/

https://superuser.com/questions/956311/continuously-detect-new-files-with-inotify-tools-within-multiple-directories-r

https://stackoverflow.com/questions/10929453/read-a-file-line-by-line-assigning-the-value-to-a-variable