/create_e2e_tests

• Create a new local test project from scratch
• Add YAML tests for a web application
• Set up authentication for a test project
• Plan what to test before writing tests

1. Always produce artifacts. Every phase writes a markdown file. Artifacts clarify your own thinking, give the user something to review, and guide later phases. When the user provides detailed requirements, use them as source material — skip questions already answered, but still produce the artifact.
2. Confirm before implementing. Present the spec (Phase 2 checkpoint) for user confirmation before spending time on browser-walking and test writing. Echo back your understanding as structured scenarios to catch mismatches early.
3. Each phase reads the previous phase's artifact. Discover feeds Specify, Specify feeds Plan, Plan feeds Implement, Implement feeds Verify. If an artifact exists from a prior run, offer to reuse it.
4. Escalate, don't loop. When something fails or is ambiguous, report it and ask the user rather than retrying silently.

1. Get project path — ask where to create the test project (e.g.,

   ./my-tests

   ). All artifacts and tests will live here. Create the

   test-specs/

   directory.
   If cloud MCP tools are available (

   SHIPLIGHT_API_TOKEN

   is set), use the

   /cloud

   skill to fetch environments and test accounts — this can pre-fill the target URL and credentials.
2. Silent scan — before asking questions, gather context from what's available:
   • Codebase: routes, components,

     package.json

     , framework
   • Git branch diff (what changed recently)
   • Existing tests (what's already covered)
   • PRDs, docs, README files
   • Cloud environments (if cloud MCP tools available)
3. Understand what to test — ask the user what they'd like to test, then ask targeted follow-up questions (one at a time, with recommendations based on your scan) to fill gaps: risk areas, user roles, authentication, data strategy, critical journeys. Skip questions the user has already answered.
4. Write

   test-strategy.md

   containing:
   • App profile: name, URL, framework, key pages/features
   • Risk profile: what matters most, what's fragile
   • Testing scope: what's in/out, user roles to cover
   • Data strategy: how test data will be created and cleaned up
   • Environment: target URL, auth method, any special setup

1. Read

   test-strategy.md

   to understand scope and priorities.
2. Generate user journey specs — for each critical journey, write:
   • Title: descriptive name (e.g., "New user signup with email verification")
   • Priority: P0 (must-have), P1 (should-have), P2 (nice-to-have)
   • Preconditions: what must be true before the test starts (Given)
   • Happy path: step-by-step actions and expected outcomes (When/Then)
   • Edge cases: at least 2 per journey (e.g., invalid input, timeout, empty state)
   • Data requirements: what test data is needed
3. Review for testing risks — scan each journey for issues that would cause flaky or incomplete tests: data dependencies, timing/async behavior, dynamic content, auth boundaries, third-party services, state isolation, environment differences. Add a Testing Notes section to each journey with identified risks and mitigations. If anything is ambiguous, ask the user (one at a time, with a recommended answer and impact statement).
4. Write

   test-spec.md

   with all journey specs.
5. Checkpoint — present a summary table for user review:

   #	Journey	Priority	Steps	Edge Cases	Risks
   1	User signup	P0	5	3	Timing
   2	...	...	...	...	...

   Ask: "Does this look right? Any journeys to add, remove, or reprioritize?"
   Wait for user confirmation before proceeding.

1. Read

   test-spec.md

   .
2. Define test file structure — map journeys to test files:

   tests/
   ├── auth.setup.ts          (if auth needed)
   ├── signup.test.yaml        (Journey 1)
   ├── checkout.test.yaml      (Journey 2)
   └── ...

3. Set implementation order — ordered by:
   • Dependencies first (auth setup before authenticated tests)
   • Then by priority (P0 before P1)
   • Then by risk (highest risk first)
4. Per-test guidance — for each test file, specify:
   • Data strategy: what data to create/use, cleanup approach
   • Wait strategy: where to use WAIT_UNTIL vs WAIT, expected loading points
   • Flakiness risks: specific things to watch for in this test
5. Write

   test-plan.md

   .
6. Checkpoint — present summary:

   Ready to implement N test files. Shall I proceed?

1. Configure AI provider — check if the test project already has a

   .env

   with an AI API key. If not, ask the user to choose a provider:

   To run YAML tests, I need an AI provider for resolving test steps. Which provider would you like to use?

   A) Google AI —

   GOOGLE_API_KEY

   (Get key) — default model:

   gemini-3.1-flash-lite-preview

   B) Anthropic —

   ANTHROPIC_API_KEY

   (Get key) — default model:

   claude-haiku-4-5

   C) OpenAI —

   OPENAI_API_KEY

   (Get key) — default model:

   gpt-5.4-mini

   D) Azure OpenAI — requires

   AZURE_OPENAI_API_KEY

   +

   AZURE_OPENAI_ENDPOINT

   — set

   WEB_AGENT_MODEL=azure:<deployment>

   E) AWS Bedrock — uses AWS credential chain — set

   WEB_AGENT_MODEL=bedrock:<model_id>

   F) Google Vertex AI — uses GCP Application Default Credentials — set

   WEB_AGENT_MODEL=vertex:<model>

   G) I already have it configured

   After the user chooses, ask for their API key and save it to the test project's

   .env

   file. For A/B/C, the model is auto-detected from the key. For D/E/F, also save

   WEB_AGENT_MODEL

   with the appropriate

   provider:model

   prefix. Optionally, the user can set

   WEB_AGENT_MODEL

   to override the default model (e.g.,

   WEB_AGENT_MODEL=claude-sonnet-4-6

   ).
2. Scaffold the project — call

   scaffold_project

   with the absolute project path. This creates

   package.json

   ,

   playwright.config.ts

   ,

   .env.example

   ,

   .gitignore

   , and

   tests/

   . Save the API key to

   .env

   .
3. Install dependencies:
   bash

   npm install
   npx playwright install chromium

4. Set up authentication (if needed) — follow the standard Playwright authentication pattern.
   Add credentials as variables in

   playwright.config.ts

   :
   ts

   {
     name: 'my-app',
     testDir: './tests/my-app',
     dependencies: ['my-app-setup'],
     use: {
       baseURL: 'https://app.example.com',
       storageState: 'tests/my-app/.auth/storage-state.json',
       variables: {
         username: process.env.MY_APP_EMAIL,
         password: { value: process.env.MY_APP_PASSWORD, sensitive: true },
         // otp_secret_key: { value: process.env.MY_APP_TOTP_SECRET, sensitive: true },
       },
     },
   },

   Standard variable names:

   username

   ,

   password

   ,

   otp_secret_key

   . Use

   { value, sensitive: true }

   for secrets. Add values to

   .env

   .
   Write

   auth.setup.ts

   with standard Playwright login code. For TOTP, implement RFC 6238 using

   node:crypto

   (HMAC-SHA1 + base32 decode) — no third-party dependency needed.
   Verify auth before proceeding. Run

   npx shiplight test --headed

   to execute the auth setup and confirm it saves

   storage-state.json

   . If it fails, escalate to the user — auth is a prerequisite for everything else.
   If the test plan involves special auth requirements (e.g., one account per test, multiple roles), confirm the auth strategy with the user before proceeding.

1. Open a browser session — call

   new_session

   with the app's

   starting_url

   .
2. Walk through the flow — use

   inspect_page

   to see the page, then

   act

   to perform each action. This captures locators from the response.
3. Capture locators — use

   get_locators

   for additional element info when needed.
4. Build the YAML — construct the

   .test.yaml

   content following the best practices below.
5. Save and validate — write the

   .test.yaml

   file, then call

   validate_yaml_test

   with the file path to check locator coverage (minimum 50% required).
6. Close the session — call

   close_session

   when done.

validate_yaml_test

test-plan.md

• Apply the specified wait strategy at loading points
• Cover the edge cases and assertions defined in the spec

npx shiplight test --headed

1. Report — tell the user which test failed and why (one sentence).
2. Classify the failure:
   • Implementation fix (wrong locator, missing wait, timing) → fix and retry.
   • Spec mismatch (app behavior differs from spec) → ask the user whether to update the spec or skip the scenario.
3. Escalate if a fix doesn't work — don't keep retrying the same approach.

1. Update

   test-spec.md

   — mark skipped scenarios with reason, add scenarios that emerged during implementation, update edge cases to reflect what was tested
2. Update

   test-plan.md

   — correct file structure, note deviations from the original plan
3. Show diff summary — tell the user what changed and why

• ACTION is the default. Capture locators via MCP tools (

  act

  ,

  get_locators

  ) during browser sessions, then write ACTION statements. ACTIONs replay deterministically (~1s).
• DRAFT is a last resort. Only use DRAFT when the locator is genuinely unknowable at authoring time. DRAFTs are slow (~5-10s each, AI resolution at runtime). Tests with too many DRAFTs are rejected by

  validate_yaml_test

  .
• VERIFY for assertions. Use

  VERIFY:

  for all assertions. Do not write assertion DRAFTs like

  "Check that the button is visible"

  .
• URL for navigation. Use

  URL: /path

  for navigation instead of

  action: go_to_url

  .
• CODE for scripting. Use

  CODE:

  for network mocking, localStorage manipulation, page-level scripting. Not for clicks, assertions, or navigation.

• intent: Click button

• intent: Click the 3rd button in the form
• intent: Click element at index 42

• intent: Click the Submit button to save the new project action: click locator: "getByRole('button', { name: 'Submit' })"

undefined

• Always resolve locators to a single element (e.g.,

  .first()

  ,

  .nth(1)

  ) to avoid Playwright strict-mode errors
• Always include

  { timeout: 5000 }

  on actions for predictable timing
• The

  intent

  is critical — it's the input for self-healing when

  js

  fails

• page

  ,

  agent

  , and

  expect

  are available in scope

• Always set a short timeout (e.g.,

  { timeout: 2000 }

  ) on

  js:

  assertions that have an AI fallback, so stale locators fall back to AI quickly instead of waiting the default 5s
• Always use

  VERIFY:

  shorthand — do not use

  action: verify

  directly
• Be aware of false negatives with

  js:

  assertions. The AI fallback only triggers when

  js

  throws (element not found, timeout). If

  js

  passes against the wrong element (stale selector matching a different element), the assertion silently succeeds — no fallback occurs. Keep

  js:

  assertions simple and specific to minimize this risk.

• Use natural language (AI) conditions for DOM-based checks (element visible, text present, page state). AI conditions self-heal against DOM changes;

  js:

  conditions are brittle and cannot auto-heal.
• Use

  js:

  conditions only for counter/state logic — e.g.,

  js: counter++ < 10

  ,

  js: retryCount < 3

  . Never use

  js:

  for DOM inspection like

  js: document.querySelector('.modal') !== null

  .
• If you need a JavaScript-based DOM check, use

  CODE:

  to evaluate it and store the result, or use

  VERIFY:

  with

  js:

  (which at least has AI fallback on failure).

• WAIT_UNTIL:

  — AI checks the condition repeatedly until met or timeout. Default timeout is 60 seconds. Each AI check takes 5–10s, so set

  timeout_seconds

  to at least 15.

• WAIT:

  — fixed-duration pause. Use

  seconds:

  to set duration.

• Put

  intent

  first in ACTION statements for readability

• xpath

  is only needed when an ACTION has neither

  locator

  nor

  js

  .
• Single-test vs Suite vs Parameters:
  • Single-test file — one isolated test, no shared state
  • Suite — tests that have sequential dependencies (e.g., test A creates a file, test B consumes it). Each test in a suite still covers one journey — the suite just guarantees execution order and shares browser state. Do NOT use suites to bundle unrelated tests.
  • Parameters — same test structure, different data inputs

• URL: /projects
• intent: Click on "My Project" action: click locator: "getByText('My Project')"
• intent: Click the Delete button action: click locator: "getByRole('button', { name: 'Delete' })"

• CODE: js: | const res = await page.request.post('/api/projects', { data: { name: 'Delete-Test-' + Date.now() } }); const project = await res.json(); save_variable('projectName', project.name);
• URL: /projects
• WAIT_UNTIL: The project list has loaded
• intent: Click on the project we just created action: click js: "await page.getByText('{{projectName}}').click()"
• intent: Click the Delete button action: click locator: "getByRole('button', { name: 'Delete' })"
• VERIFY: The project is no longer visible in the list

undefined

• intent: Log in
• intent: Navigate to settings
• intent: Change display name
• VERIFY: Name updated
• intent: Navigate to account
• intent: Delete account
• VERIFY: Account deleted

• URL: /settings
• intent: Clear the display name field and type "New Name" action: fill locator: "getByLabel('Display name')" value: "New Name"
• intent: Click Save action: click locator: "getByRole('button', { name: 'Save' })"
• VERIFY: Success message "Settings saved" is visible

• URL: /account

... focused on deletion only

undefined

• VERIFY: js: | const el = await page.locator('.btn-primary'); await expect(el).toHaveClass(/disabled/); await expect(el).toHaveAttribute('data-state', 'submitted');

• VERIFY: The Submit button is disabled js: | await expect(page.getByRole('button', { name: 'Submit' })) .toBeDisabled({ timeout: 2000 });

undefined

• VERIFY: Page title is "Dashboard"
• VERIFY: Welcome message shows username
• VERIFY: Sidebar has 5 menu items
• VERIFY: Footer shows current year
• VERIFY: Avatar image is loaded
• VERIFY: Notification bell is visible

• VERIFY: Dashboard page shows the welcome message with the user's name

undefined

• intent: Enter card number in Stripe iframe
• intent: Click Stripe's pay button
• VERIFY: Stripe shows success checkmark

• CODE: js: | await page.route('**/api/payments', route => route.fulfill({ status: 200, json: { status: 'succeeded', id: 'pi_mock' } }) );
• intent: Click the Pay button action: click locator: "getByRole('button', { name: 'Pay' })"
• VERIFY: Order confirmation page shows "Payment successful"

undefined

• intent: Type "Test User" into the name field action: fill locator: "getByLabel('Name')" value: "Test User"

• CODE: js: "save_variable('testName', 'Test-User-' + Date.now());"
• intent: Type the generated name into the name field action: fill locator: "getByLabel('Name')" text: "{{testName}}"

undefined

• URL: /projects/new
• intent: Type project name
• intent: Select team
• intent: Click Create
• WAIT_UNTIL: Project page loads

• CODE: js: | const res = await page.request.post('/api/projects', { data: { name: 'Seed-' + Date.now(), team: 'engineering' } }); const { slug } = await res.json(); save_variable('projectSlug', slug);
• URL: /projects/{{projectSlug}}/settings
• WAIT_UNTIL: Settings page has loaded

• WAIT: Wait for data to load seconds: 5
• VERIFY: The table shows results

• WAIT_UNTIL: The data table has at least one row visible timeout_seconds: 30

• intent: Type search query action: fill locator: "getByRole('searchbox')" value: "test"
• WAIT: Wait for debounce to fire seconds: 1
• VERIFY: Search suggestions are visible

Rule of thumb: if the delay is **predictable and under 5s** (animation, debounce, transition), use `WAIT:`. If the delay is **unpredictable** (API call, data loading, file processing), use `WAIT_UNTIL:`.

• URL: /search
• intent: Type a query that returns no results action: fill locator: "getByRole('searchbox')" value: "zzz_no_match_zzz"
• intent: Submit the search action: click locator: "getByRole('button', { name: 'Search' })"
• VERIFY: Empty state message "No results found" is displayed
• VERIFY: The search box still contains the query (user can refine)

undefined

• Use unique, per-test data instead of shared fixtures
• Avoid tests that change global configuration
• If a test must modify shared state, document it and mark it for serial execution

• Timing flake? → Add a proper

  WAIT_UNTIL:

  for the right condition
• Data flake? → Use unique test data, add proper cleanup
• Order flake? → The test has a hidden dependency on another test — make it self-contained
• Environment flake? → Mock the unstable external service

• ACTION statements with locators replay ~10x faster than DRAFTs. Always prefer ACTIONs.
• Use

  inspect_page

  to understand page state. Always read the DOM file first — it provides element indices needed for

  act

  and consumes far fewer tokens. Only view the screenshot when you specifically need visual information (layout, colors, images), as screenshots consume significantly more tokens than DOM.
• Run a specific project's tests with:

  npx shiplight test --headed my-saas-app/

• The

  .env

  file is auto-discovered by

  shiplightConfig()

  — no manual dotenv setup needed.