Marketplace Engineering Two-Sided Search and Recsys Planning Best Practices

When to Apply

• Planning a new marketplace retrieval project from scratch
• Reviewing an existing retrieval system that feels stale, unfair, or unpersonalised
• Designing the OpenSearch index mapping, analyzers, or query DSL
• Choosing retrieval primitives per product surface (search, recs, hybrid, curated)
• Deciding which search quality metrics to track and dashboard
• Running the weekly search-quality review ritual
• Diagnosing a silent regression in ranking, coverage, or zero-result rate
• Deciding when a retrieval problem is actually a personalisation problem

marketplace-personalisation

Living Context

• gotchas.md

  (in this skill folder) — append-only diagnostic lessons. Every gotcha has a date and a short description of what surprised the team and how it was resolved.
• Decisions log (maintained in the product repo, typically

  decisions/*.md

  ) — every ranking change, schema tweak, and synonym edit recorded with its hypothesis, offline and online evidence, ship criterion, outcome, and rollback path. See rule

  plan-maintain-a-decisions-log

  .
• Golden query set (frozen per eval cycle, committed to the product repo) — the reference set of queries against which every ranking change is offline-evaluated before an online test. See rule

  plan-version-the-golden-set

  .

Rule Categories

intent-

arch-

index-

plan-

query-

retrieve-

rank-

blend-

measure-

monitor-

Quick Reference

1. Problem Framing and User Intent (CRITICAL)

• intent-map-queries-to-intent-classes

  — classify before retrieving

• intent-separate-known-item-from-discovery

  — different failure modes, different strategies

• intent-audit-live-query-logs-first

  — design from real data, not imagined data

• intent-distinguish-transactional-from-exploratory

  — precision vs diversity

• intent-reject-one-search-for-everything

  — per-surface query shapes

• intent-treat-no-search-as-first-class-choice

  — curated is a legitimate answer

2. Surface Taxonomy and Architecture (CRITICAL)

• arch-map-surface-to-retrieval-primitive

  — a single-source-of-truth routing table

• arch-split-candidate-generation-from-ranking

  — two-stage pipelines

• arch-design-zero-result-fallback

  — declare fallback owner per surface

• arch-design-for-cold-start-from-day-one

  — cold start is permanent, not bootstrap

• arch-avoid-mono-stack-retrieval

  — diversify primary dependencies

• arch-route-surfaces-deliberately

  — every routing decision recorded

3. Index Design and Mapping (HIGH)

• index-design-mappings-conservatively

  — reindex is expensive

• index-use-keyword-and-text-as-multi-fields

  — full-text plus exact match

• index-match-index-and-query-time-analyzers

  — tokens must agree

• index-use-language-analyzers-for-language-fields

  — language-aware stemming

• index-separate-searchable-from-display-fields

  — index only what you search

• index-use-index-templates-for-consistency

  — prevent mapping drift

• index-stream-listing-updates-via-cdc

  — freshness in seconds, not hours

4. Planning and Improvement Methodology (HIGH)

• plan-audit-before-you-build

  — instrumentation gate on kick-off

• plan-build-golden-query-set-first

  — the first artefact, not the last

• plan-find-bottleneck-before-optimising

  — theory of constraints

• plan-maintain-a-decisions-log

  — living context across team changes

• plan-version-the-golden-set

  — frozen per eval cycle

• plan-handoff-to-personalisation-skill

  — recognise the boundary

5. Query Understanding (MEDIUM-HIGH)

• query-normalise-before-anything-else

  — canonical string in

• query-use-language-analyzers-for-stemming

  — double-digit recall wins

• query-curate-synonyms-by-domain

  — domain vocabulary not thesaurus

• query-use-fuzzy-matching-for-typos

  — 10-15% of queries have typos

• query-classify-before-routing

  — single-pass classifier

• query-build-autocomplete-on-separate-index

  — latency isolation

6. Retrieval Strategy (MEDIUM-HIGH)

• retrieve-use-filter-clauses-for-exact-matches

  — filter cache wins

• retrieve-use-bool-structure-deliberately

  — must vs should vs filter

• retrieve-run-expensive-signals-in-rescore

  — rescore window limits cost

• retrieve-combine-bm25-and-knn-via-hybrid-search

  — lexical plus semantic

• retrieve-paginate-with-search-after

  — constant-cost deep pagination

• retrieve-choose-embedding-model-deliberately

  — re-embedding is expensive

7. Relevance and Ranking (MEDIUM-HIGH)

• rank-tune-bm25-parameters-last

  — upstream levers first

• rank-use-function-score-for-business-signals

  — explicit named functions

• rank-deploy-ltr-only-after-golden-set-exists

  — supervised learning needs labels

• rank-apply-diversity-at-rank-time

  — after scoring, not before

• rank-normalise-scores-across-retrieval-primitives

  — comparable scales

8. Search and Recommender Blending (MEDIUM)

• blend-use-search-alone-for-specific-intent

  — precision queries

• blend-combine-search-and-personalisation-scores

  — normalised weighted sum

• blend-keep-hybrid-blending-explainable

  — traceable results

• blend-never-return-zero-results

  — guaranteed cascade to non-empty

9. Measurement and Experimentation (MEDIUM)

• measure-define-session-success-per-surface

  — one definition per surface

• measure-track-ndcg-mrr-zero-result-rate

  — three metrics for one picture

• measure-track-reformulation-rate-as-failure-signal

  — cheapest failure metric

• measure-use-click-models-for-implicit-judgments

  — scale beyond human judges

• measure-run-interleaving-as-cheap-ab-proxy

  — 10x less sample needed

10. Instrumentation, Dashboards and Decision Triggers (MEDIUM)

• monitor-log-every-query-with-full-context

  — structured replayable events

• monitor-scrub-pii-from-query-logs

  — redact before warehouse ingestion

• monitor-build-search-health-dashboard

  — threshold lines, colour bands

• monitor-alert-on-decision-triggers

  — quality metrics, not error rates

• monitor-track-ranking-stability-churn

  — RBO churn as leading indicator

• monitor-run-weekly-search-quality-review

  — calendar-driven ritual

Planning and Improving

• references/playbooks/planning.md

  — Plan a new marketplace retrieval system from scratch. Nine-step workflow from intent audit through the first A/B-tested online lift, with explicit exit criteria per step.

• references/playbooks/improving.md

  — Diagnose and improve an existing retrieval system. Decision tree that walks through telemetry, index freshness, coverage, baseline gap, cold start, segment regressions, and algorithm iteration in that order, with hand-off points to

  marketplace-personalisation

  when the bottleneck is personalisation-specific.

How to Use

• Read

  references/_sections.md

  for category structure and cascade rationale.
• Read

  gotchas.md

  for diagnostic lessons accumulated from prior incidents.
• Read

  references/playbooks/planning.md

  to plan a new system.
• Read

  references/playbooks/improving.md

  to diagnose an existing one.
• Read individual rule files when a specific task matches the rule title.
• Use

  assets/templates/_template.md

  to author new rules as the skill grows.

Related Skills

• marketplace-personalisation

  — The companion skill covering AWS Personalize implementation, impression tracking, schema design, two-sided matching, feedback loops, and the personalisation-specific diagnostic playbook. Hand off to this skill when the diagnostic identifies a personalisation-specific bottleneck.

Reference Files