Resilience Review

When to use

/resilience-review

• Before launching a user-facing feature
• After adding new API integrations or data sources
• When reliability is critical (healthcare, finance, e-commerce checkout)
• After production incidents caused by unhandled errors
• When moving from prototype to production quality

Standards Referenced

• Google SRE Principles — Error budgets, graceful degradation
• Netflix Chaos Engineering Principles — Verify steady state, inject real-world failures
• OWASP Error Handling — Secure and user-friendly error responses
• Nielsen Norman Group — Error message usability heuristics

Phase Overview

Phase 1: EDUCATE   → Why resilience matters and what we test
Phase 2: SCOPE     → Map failure points, dependencies, critical flows
Phase 3: ANALYZE   → Browser-based fault injection and edge case testing
Phase 4: REPORT    → Findings with evidence and user impact assessment
Phase 5: REMEDIATE → Fix guidance + YAML regression tests

Phase 1: Educate

Why this matters: Users don't experience your app in ideal conditions. 53% of mobile visits are abandoned if a page takes >3 seconds. Error pages with no guidance increase support tickets 5x. A blank screen is the worst possible failure mode — it tells the user nothing and offers no recovery path. Resilient apps maintain trust even when backend systems fail.

Phase 2: Scope

Gather context

1. Auto-detect from codebase:
   • API calls and their endpoints
   • Error boundary components (React ErrorBoundary, Vue errorHandler)
   • Loading state implementations (spinners, skeletons, suspense)
   • Empty state components
   • Retry logic / error recovery patterns
   • Offline support (service workers, cache strategies)
   • Third-party service dependencies
2. Ask the user (one at a time):
   • Target URL: Where is the app running?
   • Critical user flows: Which flows must never show a blank screen? (auto-detect from routes)
   • Key API dependencies: Which APIs does the frontend depend on? (auto-detected)
   • Known fragile areas: Any pages/features that break frequently? (optional)
3. Map failure points:
   • API endpoints the frontend calls (and what happens if each fails)
   • Third-party dependencies (CDN, auth provider, analytics, maps, payment)
   • Data-dependent UI (what shows when data is empty, missing, or malformed)
   • User input edge cases (long text, special characters, empty submissions)

Phase 3: Analyze

new_session

record_evidence: true

Category A: Error Handling (ERR)

get_browser_console_logs

CODE

page.route()

get_browser_console_logs

// Example: Mock API 500 error
await page.route('**/api/**', route => {
  route.fulfill({ status: 500, body: JSON.stringify({ error: 'Internal Server Error' }) });
});

Category B: Graceful Degradation (DEG)

navigator.userAgent

page.route()

Category C: Empty & Edge States (EDGE)

<script>

"'&<>

act

Category D: API Contract & Data Handling (API)

page.route()

Category E: Recovery & User Communication (REC)

Phase 4: Report

shiplight/reports/resilience-review-{date}.md

# Resilience Review Report
**Date:** {date}
**URL:** {url}
**Critical flows tested:** {list}
**API dependencies tested:** {count}
**Failure scenarios simulated:** {count}

## Overall Score: {X}/10 | Confidence: {X}%

## Score Breakdown
| Category | Score | Findings |
|----------|-------|----------|
| Error Handling (ERR) | 5/10 | 2 critical, 1 high |
| Graceful Degradation (DEG) | 6/10 | 1 high, 2 medium |
| Empty & Edge States (EDGE) | 4/10 | 1 critical, 3 high |
| API Contract (API) | 7/10 | 1 high, 1 medium |
| Recovery (REC) | 3/10 | 2 high, 1 medium |

## Failure Matrix
| Failure Scenario | Expected Behavior | Actual Behavior | Status |
|-----------------|-------------------|-----------------|--------|
| API returns 500 | Error message + retry | Blank screen | FAIL |
| Network timeout | Loading → timeout message | Infinite spinner | FAIL |
| Empty data set | "No results" message | Blank page | FAIL |
| ... | | | |

## Findings
(structured findings with evidence, screenshots of failure states)

Confidence Scoring

• 90-100%: Fault injected and failure behavior verified in browser
• 70-89%: Code analysis shows missing error handling, not validated at runtime
• 50-69%: Pattern-based assessment (e.g., no error boundary detected)
• Below 50%: Don't report

Phase 5: Remediate

1. Fix guidance (example)

#### ERR-01: API error shows blank screen instead of error message
**Impact:** Users see empty page, think app is broken, leave
**File:** src/pages/Dashboard.tsx:45
**Current:** `const data = await fetch('/api/data').then(r => r.json())`
**Problem:** No error handling — fetch throws on network error, .json() throws on non-JSON response
**Fix:**
- Wrap in try/catch
- Add error state: `const [error, setError] = useState(null)`
- Render error UI with retry button
- Add React Error Boundary as fallback

2. YAML regression test

- name: err-01-api-error-shows-message
  description: Verify API failure shows user-friendly error message instead of blank screen
  severity: critical
  standard: UX-Error-Handling
  steps:
    - CODE: |
        await page.route('**/api/data**', route => {
          route.fulfill({
            status: 500,
            contentType: 'application/json',
            body: JSON.stringify({ error: 'Internal Server Error' })
          });
        });
    - URL: /dashboard
    - WAIT_UNTIL: Page has finished attempting to load data
      timeout_seconds: 15
    - VERIFY: An error message is visible explaining that data could not be loaded
    - VERIFY: A retry button or recovery action is available to the user
    - VERIFY: The page is NOT blank — navigation and header are still visible

shiplight/tests/resilience-review.test.yaml

Tips

• Use

  page.route()

  in CODE blocks — it's the primary tool for fault injection
• Test the most critical user flows first (checkout, signup, core feature)
• A blank screen is always a CRITICAL finding — it's the worst failure mode
• Check

  get_browser_console_logs

  for uncaught promise rejections — they indicate missing error handling
• Edge case testing (EDGE category) often reveals the most bugs per minute spent
• Close session with

  close_session

  and use

  generate_html_report

  for evidence