Loading...
Loading...
Distributed systems patterns for locking, resilience, idempotency, and rate limiting. Use when implementing distributed locks, circuit breakers, retry policies, idempotency keys, token bucket rate limiters, or fault tolerance patterns.
npx skill4agent add yonatangross/orchestkit distributed-systemsrules/| Category | Rules | Impact | When to Use |
|---|---|---|---|
| Distributed Locks | 3 | CRITICAL | Redis/Redlock locks, PostgreSQL advisory locks, fencing tokens |
| Resilience | 3 | CRITICAL | Circuit breakers, retry with backoff, bulkhead isolation |
| Idempotency | 3 | HIGH | Idempotency keys, request dedup, database-backed idempotency |
| Rate Limiting | 3 | HIGH | Token bucket, sliding window, distributed rate limits |
| Edge Computing | 2 | HIGH | Edge workers, V8 isolates, CDN caching, geo-routing |
| Event-Driven | 2 | HIGH | Event sourcing, CQRS, transactional outbox, sagas |
# Redis distributed lock with Lua scripts
async with RedisLock(redis_client, "payment:order-123"):
await process_payment(order_id)
# Circuit breaker for external APIs
@circuit_breaker(failure_threshold=5, recovery_timeout=30)
@retry(max_attempts=3, base_delay=1.0)
async def call_external_api():
...
# Idempotent API endpoint
@router.post("/payments")
async def create_payment(
data: PaymentCreate,
idempotency_key: str = Header(..., alias="Idempotency-Key"),
):
return await idempotent_execute(db, idempotency_key, "/payments", process)
# Token bucket rate limiting
limiter = TokenBucketLimiter(redis_client, capacity=100, refill_rate=10)
if await limiter.is_allowed(f"user:{user_id}"):
await handle_request()| Rule | File | Key Pattern |
|---|---|---|
| Redis & Redlock | | Lua scripts, SET NX, multi-node quorum |
| PostgreSQL Advisory | | Session/transaction locks, lock ID strategies |
| Fencing Tokens | | Owner validation, TTL, heartbeat extension |
| Rule | File | Key Pattern |
|---|---|---|
| Circuit Breaker | | CLOSED/OPEN/HALF_OPEN states, sliding window |
| Retry & Backoff | | Exponential backoff, jitter, error classification |
| Bulkhead Isolation | | Semaphore tiers, rejection policies, queue depth |
| Rule | File | Key Pattern |
|---|---|---|
| Idempotency Keys | | Deterministic hashing, Stripe-style headers |
| Request Dedup | | Event consumer dedup, Redis + DB dual layer |
| Database-Backed | | Unique constraints, upsert, TTL cleanup |
| Rule | File | Key Pattern |
|---|---|---|
| Token Bucket | | Redis Lua scripts, burst capacity, refill rate |
| Sliding Window | | Sorted sets, precise counting, no boundary spikes |
| Distributed Limits | | SlowAPI + Redis, tiered limits, response headers |
| Rule | File | Key Pattern |
|---|---|---|
| Edge Workers | | V8 isolate constraints, Web APIs, geo-routing, auth at edge |
| Edge Caching | | Cache-aside at edge, CDN headers, KV storage, stale-while-revalidate |
| Rule | File | Key Pattern |
|---|---|---|
| Event Sourcing | | Event-sourced aggregates, CQRS read models, optimistic concurrency |
| Event Messaging | | Transactional outbox, saga compensation, idempotent consumers |
| Decision | Recommendation |
|---|---|
| Lock backend | Redis for speed, PostgreSQL if already using it, Redlock for HA |
| Lock TTL | 2-3x expected operation time |
| Circuit breaker recovery | Half-open probe with sliding window |
| Retry algorithm | Exponential backoff + full jitter |
| Bulkhead isolation | Semaphore-based tiers (Critical/Standard/Optional) |
| Idempotency storage | Redis (speed) + DB (durability), 24-72h TTL |
| Rate limit algorithm | Token bucket for most APIs, sliding window for strict quotas |
| Rate limit storage | Redis (distributed, atomic Lua scripts) |
# LOCKS: Never forget TTL (causes deadlocks)
await redis.set(f"lock:{name}", "1") # WRONG - no expiry!
# LOCKS: Never release without owner check
await redis.delete(f"lock:{name}") # WRONG - might release others' lock
# RESILIENCE: Never retry non-retryable errors
@retry(max_attempts=5, retryable_exceptions={Exception}) # Retries 401!
# RESILIENCE: Never put retry outside circuit breaker
@retry # Would retry when circuit is open!
@circuit_breaker
async def call(): ...
# IDEMPOTENCY: Never use non-deterministic keys
key = str(uuid.uuid4()) # Different every time!
# IDEMPOTENCY: Never cache error responses
if response.status_code >= 400:
await cache_response(key, response) # Errors should retry!
# RATE LIMITING: Never use in-memory counters in distributed systems
request_counts = {} # Lost on restart, not shared across instances| Resource | Description |
|---|---|
| scripts/ | Templates: lock implementations, circuit breaker, rate limiter |
| checklists/ | Pre-flight checklists for each pattern category |
| references/ | Deep dives: Redlock algorithm, bulkhead tiers, token bucket |
| examples/ | Complete integration examples |
cachingbackground-jobsobservability-monitoringerror-handling-rfc9457auth-patterns