Turn the user's workload into a reproducible end-to-end comparison. Deliver a runnable harness when implementation is requested, or an actionable plan when planning is requested. Reuse an existing harness before creating another one. Do not invent a Parallel CLI eval command.

For product selection without a benchmark, use ; for replacing an existing search integration, use . Use this skill to measure the resulting quality, cost, and latency.

Read the public evaluation guide, Search best practices, and Extract best practices before implementing requests. Check current modes and pricing; record the documentation date and configuration rather than embedding price or latency promises.

Establish the application, question types, providers, existing agent/model, verified answers, and quality/cost/latency target. Infer these from supplied code and data; ask only for missing decisions that change the comparison. Prepare offline artifacts while credentials or a live-run budget remain unresolved. Use existing authorization for paid calls; otherwise establish scope and spend limits before execution. Never request keys in chat.

Choose the product that matches the application:

Always set Search explicitly for REST requests: omitting it selects , changing latency and price. Unless the user specifies otherwise, start with for interactive workloads or for harder research, then test another relevant mode. The evaluation guide still recommends ; record this discrepancy and the chosen rationale rather than silently inheriting that starting point. Honor workload constraints such as language support and endpoint availability. For Search MCP, consult its current routing and override rules: pin the mode for authenticated calls where supported; anonymous calls reject overrides, so record the actual server-selected mode or require authentication for a controlled mode comparison. Keep the model fixed within a comparison; a different model/mode pairing is a separate experiment.

Use references/eval-contract.md when creating dataset records, run manifests, traces, grading output, and reports. Resolve this reference relative to this skill's directory.

Recommend a cost-parity comparison: align result count, content per page, product tier, and reasoning level where supported, and account for total Search/Extract/model cost including input and output tokens. Start with documented defaults as the baseline; any parity overrides need a recorded rationale and a separate configuration. Report requested and actual result counts and content volumes, residual mismatches, and settings that cannot be equated. Do not treat equal API-call counts as equal cost or imply parity when it was not achieved.

Use the same research prompt across providers and settings; change tool availability and executor budgets separately. For public benchmarks, use the guide's research prompt and official grader where available. Record output-format overrides and label altered benchmark protocols as adapted evaluations.

Inspect a small fixed pilot: query quality, warnings, excerpt delivery, tool limits, concurrency, and trace completeness. Fix integration faults and rerun affected configurations; do not tune during a scored run. Freeze settings before evaluating held-out questions. Run each configuration at least three times and report variation.

Flag every failure to the user with affected trial IDs and impact: rate limits, HTTP errors, timeouts, partial results, token truncation, dropped results, and endpoint fallback (including → ). Record each occurrence even if a retry succeeds. Never silently fall back or count a Parallel-side failure as a retrieval miss. Surface failures during execution and include their counts and trace references in the final report; preserve partial successes and the declared scoring denominator.

Grade final answers by task shape: factual correctness and citation support; list precision and recall; structured field accuracy; or an explicit research rubric. Require judge reasoning. For custom rubrics, rank honest inability to retrieve above confident misinformation. Hand-check at least 10% of judged trials, covering successes and failures. Keep citation audits separate from official benchmark scores.

Classify failures from traces: no search call, synthesis failure despite sufficient evidence, provider error, or retrieval miss despite a reasonable query. Do not attribute missing excerpts or other harness faults to retrieval quality. Distinguish account rate limits from provider capability.

Include failures and timeouts in primary results under the declared scoring policy. Bound retries and include all their time and cost. Report Search/Extract/model charges, tokens, calls, budget exhaustion, and search and end-to-end latency at p50/p95/p99. Time the whole task by wall clock, including query reformulation and answer generation; do not sum concurrent call durations. Cost per resolved task includes spend on unsuccessful trials; report it as undefined if nothing resolves.

Deliver the reproducible configuration, dataset and trace locations, grading rationale, failure breakdown, and quality versus cost/latency comparisons. Recommend configurations on the measured Pareto frontier that meet the user's targets, with uncertainty and workload slices. If calls were not run, label the deliverable as an unexecuted plan or harness and state what remains needed.

After results, propose 2–3 follow-up configurations tied to observed failures or bottlenecks: retrieval misses → ; thin excerpts → enable or increase Extract use; high latency → ; high model-token cost → a smaller content budget. Change one factor per comparison, record the rationale, and use a new frozen manifest. Present these as proposals, not automatically launched runs; execute only when the user requests them within the authorized scope and spend.