AWS S3 挂载 EC2 实战：s3fs-fuse 与实时同步监控

背景

业务需要将 S3 对象存储作为 EC2 的扩展数据盘使用，要求：

• 稳定的文件系统挂载
• 增量同步和实时同步
• 错误处理和日志记录
• 自动重启和监控

成本对比

结论：纯存储成本 S3 比 gp3 便宜约 70%，但请求费贵；生产环境推荐 gp3 热 + S3 冷分层架构。

IAM 权限设计

策略：最小权限原则

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3BucketAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::qa-code-analysis-s3-xxx",
            "Condition": {
                "StringEquals": {
                    "aws:RequestedRegion": "ap-east-1",
                    "ec2:SourceInstanceARN": "arn:aws:ec2:ap-east-1:xxx:instance/i-xxx"
                }
            }
        },
        {
            "Sid": "S3ObjectOperations",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::qa-code-analysis-s3-xxx/*",
            "Condition": {
                "StringEquals": {
                    "ec2:SourceInstanceARN": "arn:aws:ec2:ap-east-1:xxx:instance/i-xxx"
                }
            }
        },
        {
            "Sid": "DenyAllOtherInstances",
            "Effect": "Deny",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::qa-code-analysis-s3-xxx",
                "arn:aws:s3:::qa-code-analysis-s3-xxx/*"
            ],
            "Condition": {
                "StringNotEquals": {
                    "ec2:SourceInstanceARN": "arn:aws:ec2:ap-east-1:xxx:instance/i-xxx"
                }
            }
        }
    ]
}

可信实体

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "aws:RequestedRegion": "ap-east-1"
                },
                "ArnEquals": {
                    "aws:SourceArn": "arn:aws:ec2:ap-east-1:xxx:instance/i-xxx"
                }
            }
        }
    ]
}

s3fs-fuse 安装

# Amazon Linux 2/CentOS/RHEL
sudo yum install -y gcc gcc-c++ make automake autoconf git fuse fuse-devel curl-devel libxml2-devel openssl-devel fuse3-devel

# 编译安装
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh && ./configure && make && sudo make install

# 验证
s3fs --version
# Amazon Simple Storage Service File System V1.97

挂载配置

手动挂载

BUCKET_NAME="qa-code-analysis-s3-xxx"
MOUNT_POINT="/s3-data"
REGION="ap-east-1"
CACHE_DIR="/tmp/s3fs-cache"

mkdir -pv $MOUNT_POINT $CACHE_DIR
chmod 755 $CACHE_DIR

s3fs $BUCKET_NAME $MOUNT_POINT     -o iam_role=auto     -o url=https://s3.$REGION.amazonaws.com     -o endpoint=$REGION     -o use_cache=$CACHE_DIR     -o allow_other     -o uid=0     -o gid=0     -o mp_umask=022     -o multireq_max=5     -o parallel_count=10     -o multipart_size=52     -o connect_timeout=10     -o readwrite_timeout=30

# 验证
mountpoint /s3-data

fstab 开机自动挂载

# /etc/fstab
s3fs#qa-code-analysis-s3-xxx /s3-data fuse   _netdev,iam_role=auto,url=https://s3.ap-east-1.amazonaws.com,  endpoint=ap-east-1,use_cache=/tmp,allow_other,uid=1000,gid=1000,  mp_umask=002 0 0

实时同步监控脚本

cat > /root/s3fs-fuse/s3-realtime-sync.sh <<'EOF'
#!/bin/bash
SOURCE_DIR="/s3-data"
S3_BUCKET="s3://qa-code-analysis-s3-xxx"
LOG_FILE="/var/log/s3-realtime-sync.log"
BATCH_SIZE=10
BATCH_TIMEOUT=30

# 安装 inotify-tools
yum install -y inotify-tools

log_message() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}

declare -a file_queue=()
last_batch_time=$(date +%s)

process_batch() {
    if [[ ${#file_queue[@]} -eq 0 ]]; then return; fi
    log_message "处理批量文件: ${#file_queue[@]} 个"
    for file_path in "${file_queue[@]}"; do
        if [[ -f "$file_path" ]]; then
            local s3_path="${file_path#$SOURCE_DIR}"
            aws s3 cp "$file_path" "$S3_BUCKET/data$s3_path" --storage-class STANDARD_IA
        fi
    done
    file_queue=()
    last_batch_time=$(date +%s)
}

add_to_queue() {
    file_queue+=("$1")
    local current_time=$(date +%s)
    if [[ ${#file_queue[@]} -ge $BATCH_SIZE ]] ||        [[ $((current_time - last_batch_time)) -ge $BATCH_TIMEOUT ]]; then
        process_batch
    fi
}

# 定时刷新
periodic_flush() {
    while true; do
        sleep $BATCH_TIMEOUT
        if [[ ${#file_queue[@]} -gt 0 ]]; then
            local current_time=$(date +%s)
            if [[ $((current_time - last_batch_time)) -ge $BATCH_TIMEOUT ]]; then
                process_batch
            fi
        fi
    done
}

log_message "开始实时监控: $SOURCE_DIR"
periodic_flush &
flush_pid=$!

cleanup() {
    log_message "停止实时监控..."
    kill $flush_pid 2>/dev/null
    process_batch
    exit 0
}
trap cleanup EXIT INT TERM

# 监控文件变化
inotifywait -m -r "$SOURCE_DIR"     -e create -e modify -e moved_to     --exclude '\.(tmp|log|swp)$'     --format '%w%f %e' | while read file_path event; do
    if [[ -f "$file_path" ]]; then
        log_message "检测到文件变化: $file_path ($event)"
        add_to_queue "$file_path"
    fi
done
EOF

chmod +x /root/s3fs-fuse/s3-realtime-sync.sh

Systemd 服务化

cat > /etc/systemd/system/s3-realtime-sync.service <<'EOF'
[Unit]
Description=S3 Real-time Sync Service
After=network.target

[Service]
Type=simple
User=root
ExecStart=/root/s3fs-fuse/s3-realtime-sync.sh
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable s3-realtime-sync.service
sudo systemctl start s3-realtime-sync.service

定时任务

# crontab
# 每小时增量同步
0 * * * * /usr/local/bin/s3-sync-enhanced.sh

# 每天凌晨2点完整同步
0 2 * * * /usr/local/bin/s3-sync-enhanced.sh

# 每5分钟检查挂载状态
*/5 * * * * /usr/local/bin/check-s3-mount.sh

日志查看

# 实时同步日志
tail -f /var/log/s3-realtime-sync.log

# 系统服务日志
journalctl -u s3-realtime-sync.service -f

# S3 访问日志
aws s3api get-bucket-logging --bucket qa-code-analysis-s3-xxx

复盘

问题根因：初期使用 Access Key 挂载，密钥轮换导致服务中断

改进措施：

• 改用 IAM Role 绑定 EC2 实例，免密钥管理
• 增加挂载状态监控，掉盘自动告警
• 批量上传优化，减少 S3 PUT 请求费用

本文首发于 wr.mrchi.cn，转载请注明出处。