Back to Details

r2-glacier-migration

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

R2 → AWS Glacier Deep Archive Migration

R2 → AWS Glacier Deep Archive 迁移

Migrate ~42.6 TB from Cloudflare R2 to AWS S3 Glacier Deep Archive.
将约42.6TB的数据从Cloudflare R2迁移至AWS S3 Glacier Deep Archive。

Cost Savings

成本节省

  • R2: ~$638/month → Glacier: ~$42/month
  • Savings: ~$596/month
  • R2：约638美元/月 → Glacier：约42美元/月
  • 节省成本：约596美元/月

Buckets

存储桶

BucketSizeObjectsStatus
pollinations-text16.96 TB1.86B🔄 In progress
pollinations-images25.64 TB221M⏳ Pending
Bucket容量对象数状态
pollinations-text16.96 TB1.86B🔄 进行中
pollinations-images25.64 TB221M⏳ 待处理

Quick Commands

快速命令

Check Status

检查状态

bash
undefined
bash
undefined

Is transfer running?

传输是否在运行？

ssh ninon "screen -ls"
ssh ninon "screen -ls"

View recent logs

查看最近日志

ssh ninon "tail -50 ~/r2-glacier/transfer-text.log"
ssh ninon "tail -50 ~/r2-glacier/transfer-text.log"

Check checkpoint

检查检查点

ssh ninon "cat /tmp/r2-glacier-pollinations-text-checkpoint.json | jq ."
undefined
ssh ninon "cat /tmp/r2-glacier-pollinations-text-checkpoint.json | jq ."
undefined

Attach to Screen (Interactive)

连接到Screen会话（交互式）

bash
ssh -t ninon "screen -r r2-text"
bash
ssh -t ninon "screen -r r2-text"

Detach: Ctrl+A, D

断开连接：Ctrl+A, D

undefined
undefined

Resume if Crashed

崩溃后恢复

bash
ssh ninon "screen -dmS r2-text bash -c '~/r2-glacier/run-text-transfer.sh; exec bash'"
bash
ssh ninon "screen -dmS r2-text bash -c '~/r2-glacier/run-text-transfer.sh; exec bash'"

File Locations

文件位置

LocationPathPurpose
Skillscripts/r2-to-glacier-streaming.jsMain transfer script
Skillscripts/package.jsonDependencies
ninon
~/r2-glacier/
Deployed script + deps
ninon
~/r2-glacier/run-text-transfer.sh
Wrapper with credentials
ninon
~/r2-glacier/transfer-text.log
Transfer log
ninon
/tmp/r2-glacier-*-checkpoint.json
Resume checkpoint
位置路径用途
Skillscripts/r2-to-glacier-streaming.js主迁移脚本
Skillscripts/package.json依赖项配置
ninon
~/r2-glacier/
已部署脚本及依赖
ninon
~/r2-glacier/run-text-transfer.sh
包含凭证的包装脚本
ninon
~/r2-glacier/transfer-text.log
传输日志
ninon
/tmp/r2-glacier-*-checkpoint.json
恢复检查点文件

Script Usage

脚本使用方法

bash
undefined
bash
undefined

Test (small batch)

测试（小批量）

node r2-to-glacier-streaming.js -b pollinations-text --batch-size 100 --max-batches 1
node r2-to-glacier-streaming.js -b pollinations-text --batch-size 100 --max-batches 1

Full transfer

完整迁移

node r2-to-glacier-streaming.js -b pollinations-text --batch-size 50000 --concurrency 30
node r2-to-glacier-streaming.js -b pollinations-text --batch-size 50000 --concurrency 30

Resume after interruption

中断后恢复

node r2-to-glacier-streaming.js -b pollinations-text --resume
node r2-to-glacier-streaming.js -b pollinations-text --resume

Dry run (count only)

试运行（仅计数）

node r2-to-glacier-streaming.js -b pollinations-text --dry-run

---
node r2-to-glacier-streaming.js -b pollinations-text --dry-run

---

AWS Setup

AWS配置

  • Bucket: 
    s3://pollinations-archive
  • Region: us-east-1
  • Storage Class: DEEP_ARCHIVE
  • 存储桶：
    s3://pollinations-archive
  • 区域：us-east-1
  • 存储类别：DEEP_ARCHIVE

Verify Archives

验证归档

bash
aws s3 ls s3://pollinations-archive/ --recursive --human-readable | head -20
bash
aws s3 ls s3://pollinations-archive/ --recursive --human-readable | head -20

Important Notes

重要注意事项

  • ⚠️ 180-day minimum on Glacier Deep Archive - don't delete R2 until verified!
  • Some objects fail with "Header overflow" (very large responses) - acceptable loss
  • Text bucket (1.86B objects) takes several days
  • R2 egress is free, AWS ingress is free
  • ⚠️ Glacier Deep Archive 有180天最短存储期 - 验证完成前请勿删除R2中的数据！
  • 部分对象因“Header溢出”（响应过大）迁移失败 - 该损失可接受
  • 文本桶（18.6亿个对象）需要数天时间完成迁移
  • R2流出流量免费，AWS流入流量免费

Start Images Bucket (After Text Complete)

启动图片桶迁移（文本桶完成后）

  1. Update 
    ~/r2-glacier/run-text-transfer.sh
    on ninon:
    bash
    # Change bucket name
node r2-to-glacier-streaming.js -b pollinations-images --batch-size 50000 --concurrency 30
  2. Start new screen:
    bash
    ssh ninon "screen -dmS r2-images bash -c '~/r2-glacier/run-images-transfer.sh; exec bash'"
  1. 在ninon上更新
    ~/r2-glacier/run-text-transfer.sh
    ：
    bash
    # 修改存储桶名称
node r2-to-glacier-streaming.js -b pollinations-images --batch-size 50000 --concurrency 30
  2. 启动新的Screen会话：
    bash
    ssh ninon "screen -dmS r2-images bash -c '~/r2-glacier/run-images-transfer.sh; exec bash'"

Tracking

跟踪记录

  • Issue: #5860
  • PR: #5861
  • 问题编号：#5860
  • 拉取请求：#5861