Host Performance Analysis

When to Use

• Before starting host optimization work -- confirms the bottleneck is real (Detection)
• As a sub-step of

  perf-analysis

  for bottleneck classification (Detection)
• When GPU utilization is suspiciously low and you need to know why (Detection)
• When throughput regressed but GPU kernel execution times are unchanged (Root Cause)
• When the gap between forward step iterations has increased (Root Cause)
• To compare inter-iteration overhead between two versions of the inference engine (Root Cause)
• When you need sub-operation granularity on inter-kernel gaps or graph coverage (Root Cause, kernel-level drill-down)
• When piecewise CUDA graph coverage is unexpectedly low (Root Cause, kernel-level drill-down)
• When multi-rank inference shows unexplained performance asymmetry (Root Cause, kernel-level drill-down)

• The regression is in individual kernel performance (use

  perf-nsight-compute-analysis

  )
• You need to profile a workload from scratch (use

  workload-instrumentation

  first)
• The issue is NCCL communication (use distributed analysis)

Prerequisites

• An nsys trace file (

  .sqlite

  or

  .nsys-rep

  ) from a TRT-LLM benchmark run
• For Root Cause comparison: two traces (baseline and target)
• Python 3 with sqlite3 support

Key Concepts

Host Overhead in LLM Inference

[inter-step gap] -> [_forward_step] -> [inter-step gap] -> [_forward_step] -> ...

Hidden vs Exposed Host Overhead

Forward Step Isolation

_forward_step

Phase Classification (Context vs Generation)

Phase 1: Detection (YES/NO Verdict)

Detection Metrics

Detection Workflow

Step 1: Input Validation

# Accept .sqlite or .nsys-rep
ls -la <trace_file>

# If .nsys-rep, export to SQLite first
nsys export -t sqlite -o <output.sqlite> <input.nsys-rep>

Step 2: Run Detection Script

python scripts/detect_host_overhead.py \
  --trace /path/to/trace.sqlite \
  --output /path/to/verdict.json

Step 3: Interpret Verdict

if aggregate_verdict == YES or context_verdict == YES or generation_verdict == YES:
    overall_verdict = YES

• If YES -> Proceed to Phase 2 (Root Cause) below, then use

  perf-host-optimization

  skill
• If NO -> Use

  perf-nsight-compute-analysis

  for kernel SOL% or

  trace-interpretation

  for full classification

Phase 2: Root Cause Analysis

Principles

1. Isolate forward steps, not the full trace. nsys traces contain warmup, JIT, model loading, and teardown.
2. Use structural kernel patterns for iteration detection. Allreduce grouping is more robust than kernel density.
3. Compare steady-state iterations. Filter to identical workload (same batch size, same ctx/gen mix).
4. Per-step metrics, not totals. Always compare per-step averages.

Root Cause Workflow

Step 1: Collect nsys Traces

nsys profile -o /path/to/trace \
  -t cuda,nvtx,osrt \
  --force-overwrite=true \
  --cuda-memory-usage=true \
  -w true \
  <benchmark_command> --num_requests 500

Step 2: Export to SQLite

nsys export --type=sqlite --force-overwrite=true -o trace.sqlite trace.nsys-rep

Step 3: Run Host Overhead Analysis

# Two-trace comparison
python scripts/analyze_host_overhead.py \
  --baseline /path/to/baseline/trace.sqlite \
  --target /path/to/target/trace.sqlite \
  --baseline-label "v1.1" \
  --target-label "main" \
  --output /path/to/output/analysis.txt

# Single-trace breakdown
python scripts/analyze_host_overhead.py \
  --baseline /path/to/trace.sqlite \
  --baseline-label "current"

Step 4: Interpret Results

1. Allreduce-based iteration detection -- confirms forward step boundaries
2. Per-step wall time comparison -- quantifies the regression
3. NVTX per-step breakdown -- identifies which host operations regressed
4. GPU kernel comparison -- confirms GPU execution is unchanged
5. CUDA API comparison -- detects kernel launch overhead changes

Reading the Output

Avg wall time per step: 3,317 us (baseline) vs 3,978 us (target)  +19.9%

Operation           | baseline (us/step) | target (us/step) | Delta    | Status
_fetch_new_requests |               36   |             270  | +234     | REGRESSION
broadcast_requests  |                -   |             250  | +250     | NEW
_update_requests    |              413   |             723  | +310     | REGRESSION

Kernels per step (launched): 6.2 (baseline) vs 21.9 (target)  +253%

Step 5: Kernel-Level Drill-Down (Optional)

• An operation has high wall time but the overhead is inside GPU dispatch, not between NVTX ranges
• You need to understand how much of the forward pass is graph-captured vs eager
• Per-layer overhead is significant and you need to map kernels to functional groups
• Multi-rank inference shows unexplained performance asymmetry

Kernel-Level Techniques

Workflow

1. Start with Inter-Kernel Gap Analysis — bucket the gap distribution to understand the dominant overhead type (graph dispatch, Python interpreter, host-device sync)
2. If piecewise graph is in use, run Eager vs Graph Classification to measure graph coverage and identify unnecessary eager kernels
3. For per-layer overhead, use Repeating-Pattern Mapping to isolate the highest-overhead functional group within a single layer
4. For multi-rank setups, run Straggler Detection if per-step wall time varies across ranks

Kernel-Level Findings to Optimization Patterns

.item()

Common Patterns and Root Causes

Pattern 1: Request Management Refactor

_fetch_new_requests

broadcast_requests

Pattern 2: Increased Kernel Launch Count

cudaLaunchKernel

Pattern 3: New Bookkeeping Operations

_write_finish_reasons

handle_additional_outputs

Pattern 4: Flashinfer JIT Warmup Masquerading as Inference

Pattern 5: Context-Only Bottleneck (Masked by Aggregate)

Pitfalls

1. shortName is an Integer ID

CUPTI_ACTIVITY_KIND_KERNEL

shortName

StringIds.id

2. NVTX textId vs text

textId

text

3. Duplicate NVTX Ranges from TP Ranks

4. Negative Inter-Step Gaps

gap = next_start - prev_end

5. Benchmark Window Selection

Handoff to Optimization

perf-host-optimization

1. Detection verdict and evidence: Which metrics crossed thresholds (M1-M5), whether host prep was confirmed (M3b+M3c), and per-phase breakdown.
2. NVTX-based triage (from Root Cause): Top regressing operations by absolute delta (us/step). Map NVTX range names to source functions -- see references/trtllm-nvtx-ranges.md.
3. Handoff data block: Include structured data from references/output-format.md (see "Handoff to Optimization" section).
4. Kernel-level findings (from drill-down, if performed): Inter-kernel gap distribution, graph coverage ratio, per-group overhead map, and straggler rank identification. Map findings to optimization patterns using the table in the Root Cause kernel-level drill-down section above.

Reference