storage-sync

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Storage Sync

存储同步

Synchronize memories between Turso (durable) and redb (cache) storage layers.

在Turso（持久化存储）与redb（缓存存储）层之间同步数据。

When to Sync

同步时机

On startup - After system initialization
Periodic - Scheduled background sync
Cache staleness - When redb appears outdated
Recovery - After storage failures
Manual trigger - When explicitly requested

启动时 - 系统初始化完成后
定期同步 - 后台定时同步
缓存过期 - 当redb缓存数据过时
故障恢复 - 存储故障修复后
手动触发 - 当明确发起同步请求时

Sync Process

同步流程

Check connection health:

rust

turso_client.ping().await?;
redb_env.check_integrity()?;

Fetch latest from Turso:

rust

let episodes = turso_client
    .query("SELECT * FROM episodes ORDER BY timestamp DESC LIMIT ?")
    .bind(max_episodes_cache)
    .await?;

Update redb cache:

rust

tokio::task::spawn_blocking(move || {
    let write_txn = redb.begin_write()?;
    let mut table = write_txn.open_table(EPISODES_TABLE)?;
    for episode in episodes {
        table.insert(episode.id.as_bytes(), episode.to_bytes())?;
    }
    write_txn.commit()?;
});

Sync patterns and embeddings if enabled

检查连接健康状态:

rust

turso_client.ping().await?;
redb_env.check_integrity()?;

从Turso获取最新数据:

rust

let episodes = turso_client
    .query("SELECT * FROM episodes ORDER BY timestamp DESC LIMIT ?")
    .bind(max_episodes_cache)
    .await?;

更新redb缓存:

rust

tokio::task::spawn_blocking(move || {
    let write_txn = redb.begin_write()?;
    let mut table = write_txn.open_table(EPISODES_TABLE)?;
    for episode in episodes {
        table.insert(episode.id.as_bytes(), episode.to_bytes())?;
    }
    write_txn.commit()?;
});

同步模式与嵌入数据（若启用）

Configuration

配置

rust

pub struct SyncConfig {
    pub max_episodes_cache: usize,  // Default: 1000
    pub batch_size: usize,          // Default: 100
    pub sync_patterns: bool,        // Default: true
    pub sync_embeddings: bool,      // Default: true
}

rust

pub struct SyncConfig {
    pub max_episodes_cache: usize,  // 默认值: 1000
    pub batch_size: usize,          // 默认值: 100
    pub sync_patterns: bool,        // 默认值: true
    pub sync_embeddings: bool,      // 默认值: true
}

Error Handling

错误处理

Error	Handling
Turso unavailable	Skip sync, log warning, retry later
redb corruption	Attempt repair, rebuild from Turso
Partial sync	Track progress, resume from last point

错误类型	处理方式
Turso不可用	跳过同步，记录警告，稍后重试
redb数据损坏	尝试修复，从Turso重建缓存
部分同步失败	跟踪同步进度，从断点恢复

Performance Tips

性能优化建议

Batch small operations
Incremental sync (only changes)
Parallel fetch with Tokio
Write-ahead preparation

批量处理小型操作
增量同步（仅同步变更数据）
使用Tokio并行获取数据
提前准备写入操作

Validation

验证步骤

After sync, verify:

Episode count matches
Latest episodes present
Pattern counts consistent
No orphaned embeddings

同步完成后，需验证:

数据记录数量匹配
最新数据已同步
模式数量一致
无孤立的嵌入数据

Troubleshooting

故障排除

Issue	Solution
Slow syncs	Reduce cache size, increase batch
redb lock errors	Use dedicated write task
Memory pressure	Stream large results, smaller batches

问题	解决方案
同步速度慢	减小缓存容量，增大批量处理大小
redb锁错误	使用独立的写入任务
内存压力大	流式处理大结果集，减小批量大小