1. Classify — use the Decision Tree to determine search dimensions (occupancy-only vs full tile search)
2. Design search space — select the matching template from

   references/kernel-type-templates.md

   ; prune to ≤ 30 configs via arch filters
3. Implement — add

   exhaustive_search

   + cache +

   ct.launch

   following the Step-by-Step Workflow; handle in-place writes with split-buffer if needed
4. Test — run correctness with autotune enabled and with

   DISABLE_AUTOTUNE=1

5. Validate — A/B benchmark against fixed best-known config; see

   references/search-strategies.md

• Occupancy-only kernels (elementwise, reduction, persistent with fixed block sizes): Quick Reference + Pitfall Checklist is sufficient — skip

  references/

  docs. For in-place kernels, also read Pitfall #1.
• Complex kernels (matmul with tunable tile sizes, FMHA, FP8 with num_ctas): Quick Reference → Decision Tree → API Reference → Step-by-Step Workflow → relevant

  references/

  docs.

parameter-space-design.md

kernel-type-templates.md

What type of kernel is this?
├── Compute-bound (matmul, GEMM, FMHA) → Does it have multiple tunable dimensions (tile sizes)?
│   ├── YES → Is it a fused multi-GEMM kernel (dual-GEMM, e.g. Linear+GLUAct)?
│   │   ├── YES → Template 9: low occupancy (1–2), conservative tiles (2× SHMEM/register pressure)
│   │   └── NO  → Full search: TILE_M × TILE_N × (TILE_K) × occupancy × num_ctas
│   │             (see matmul/FMHA templates in kernel-type-templates.md)
│   └── NO  → Occupancy-only search: [1, 2, 4, 8]
│             (see Quick Reference above)
├── Balanced (LayerNorm, reduction + compute) →
│   Occupancy-only search: [1, 2, 4, 8]
│   Expected benefit: 2-15%
└── Memory-bound (CE Loss, pure elementwise) →
    Occupancy-only search: [1, 2, 4, 8]
    Expected benefit: 0-15% (varies by kernel; zero-cost after tuning)

• num_ctas

  has zero benefit:

  num_ctas > 1

  enables TMA multicast, where multiple CTAs share tile data in shared memory (e.g., matmul A/B tiles reused across CTAs). Memory-bound kernels use per-element

  ct.gather

  /

  ct.scatter

  with no tile reuse — multi-CTA cooperation adds overhead with no data sharing benefit.
• Tile sizes are pre-determined: BLOCK_SIZE for memory-bound kernels is determined by offline sweep (e.g., 1024 is globally optimal on B200 across [256, 512, 1024, 2048, 4096, 8192]). This is a constant, not a runtime tunable.
• Occupancy is the only effective knob: Higher occupancy lets the GPU hide memory latency by switching to another CTA while one is stalled on a memory request.

Evidence — CE Loss experiment: A 12-config search (occupancy × num_ctas) on Cross-Entropy Loss yielded only 2.5% gain (0.79x → 0.81x vs Triton). The

num_ctas

dimension contributed nothing; the result was reverted because compilation cost outweighed the marginal benefit. Occupancy-only (4 configs) achieves the same result at 3x less compilation time.

• The tune-once/cache/launch pattern has zero runtime overhead after the first call
• The search space is tiny (4 configs, ~2-4s compilation)
• Even small improvements have value at scale

⚠️ Deprecated API:

cuda.tile_experimental.autotune_launch()

(aka

ct_experimental.autotune_launch

) is deprecated and should NOT be used. It combines search + launch in one call with random sampling, which produces less reproducible results and worse config selection compared to

exhaustive_search

. Always use

cuda.tile.tune.exhaustive_search

(the current API below) with explicit caching and

ct.launch

.

@dataclass
class TuningResult[T]:
    best: Measurement       # best config + timing (mean_us, error_margin_us, num_samples)
    successes: Sequence[Measurement]   # all successful configs (sorted by performance)
    failures: Sequence[tuple[T, str, str]]  # (config, exception_type, message)

• Exhaustive: evaluates ALL configs in order — no random sampling, no skipped configs
• Search only: does not perform the final production launch — it executes trial runs internally for benchmarking, but you call

  ct.launch

  separately for the actual production invocation
• No built-in cache: you manage caching explicitly (see tune-once/cache/launch pattern)
• Deterministic: same search space always produces the same evaluation order

• First call: runs

  exhaustive_search

  to find the best config (~2-30s depending on space size)
• Subsequent calls: uses cached config with

  ct.launch

  — zero overhead (identical to a fixed

  ct.launch

  )

_cache = {}

def run_kernel_autotuned(x, ...):
    stream = torch.cuda.current_stream()
    cache_key = (x.shape, x.dtype, str(x.device))

    if cache_key not in _cache:
        configs = list(_my_autotune_configs())
        result = exhaustive_search(
            configs, stream,
            grid_fn=lambda cfg: ...,
            kernel=my_kernel,
            args_fn=lambda cfg: ...,
            hints_fn=lambda cfg: {"occupancy": cfg.occupancy},
        )
        best_cfg = result.best.config
        tuned_kernel = my_kernel.replace_hints(occupancy=best_cfg.occupancy)
        _cache[cache_key] = (best_cfg, tuned_kernel)  # cache BOTH config and compiled kernel

    cfg, tuned_kernel = _cache[cache_key]
    grid = compute_grid(cfg)
    ct.launch(stream, grid, tuned_kernel, (x, ...))

ct.launch

_cache[cache_key]

⚠️ Critical: always cache the tuned kernel object, not just the config.

replace_hints()

returns a new kernel object with its own independent JIT cache. Calling it on every invocation triggers recompilation each time, degrading performance by 100–500×. Call

replace_hints()

once after

exhaustive_search

, store the returned kernel in the cache alongside the config, and reuse it directly on the fast path. See Pitfall #7.

• occupancy

  : Number of CTAs per SM. Higher occupancy = more parallelism but less shared memory per CTA.

• num_ctas

  : Number of CTAs in a CGA (Cooperative Group Array). Used for multi-CTA cooperation (e.g., TMA multicast). Only supported on sm90+.

undefined

1. Classify the kernel using the decision tree above.
   • VERIFY: You know whether this is occupancy-only or requires tile-size tuning.
2. Remove hardcoded hints from decorator (strongly recommended): If the kernel currently has hardcoded hints in its decorator (e.g.

   @ct.kernel(occupancy=2, num_ctas=1)

   ), remove those fixed hints and change to bare

   @ct.kernel

   before adding autotuning. While

   replace_hints

   does correctly override decorator values at runtime, leaving them creates a silent fallback trap: if any code path (e.g.,

   DISABLE_AUTOTUNE

   , error handling, or a future refactor) skips

   replace_hints

   , the decorator's fixed hints are used instead of the autotuned values — and this produces no error, just silently worse performance. Removing them makes the failure mode explicit (missing hints → compiler defaults) rather than silent (wrong fixed hints used).
   • VERIFY: The

     @ct.kernel

     decorator has no

     occupancy=

     or

     num_ctas=

     arguments before proceeding. Use bare

     @ct.kernel

     instead.
3. Check for in-place writes: If the kernel modifies input tensors in-place, you MUST use the split-buffer pattern during

   exhaustive_search

   — see Pitfall #1.
   • VERIFY: Either the kernel is not in-place, or you have added a split-buffer scratch tensor for the search phase.
4. Select the template from

   kernel-type-templates.md

   based on kernel type.
5. Design the search space following

   parameter-space-design.md

   :
   • Start from reference configs, not from scratch. Clone configs from existing production kernels of the same type (e.g.,

     ops/cutile/matmul.py

     for GEMM) and adapt. For GEMM-class kernels,

     nvMatmulHeuristics

     can suggest 8-16 high-quality candidates that reach 96-99% peak performance — see

     parameter-space-design.md

     for details.
   • Detect the current GPU architecture with

     torch.cuda.get_device_capability()

     .
   • Target one architecture at a time. Generate configs only for the detected arch. Do NOT add branches for other architectures — they cannot be tested on this machine and untested code paths are unreliable. If multi-arch support is needed later, add it in a separate pass on the appropriate hardware.
   • Identify tunable parameters (tile sizes, occupancy, num_ctas)
   • Ensure the search space includes the original fixed config (or an equivalent). This guarantees that the autotuned result is at least as good as the original — no performance regression is possible.
   • If the generated set exceeds 30, apply tile size filters and pruning rules to reduce it to ≤ 30
   • VERIFY: Total configs ≤ 30 (hard limit: CuTile compilation is heavy, >30 configs will timeout).
6. Implement the tune-once/cache/launch pattern:
   • Define a

     _cache

     dict at module level
   • Define a cache key that captures all parameters affecting optimal config (shapes, dtypes, device, any flags like

     is_causal

     ). ⚠️ Use

     str(x.device)

     not

     x.device

     in the cache key —

     torch.device

     objects are not reliably hashable and can cause

     TypeError: unhashable type

     at runtime. Always convert to string:

     cache_key = (..., x.dtype, str(x.device))

     . Tip: For GEMM-class kernels, round dimensions to the next power of 2 in the cache key (e.g.,

     cache_key = (next_pow2(M), next_pow2(N), next_pow2(K), dtype, str(device))

     ) to reduce unique key count and avoid re-tuning for similar shapes.
   • Call

     exhaustive_search(list(configs), ...)

     only when cache misses
   • Store

     result.best.config

     in cache
   • Use

     kernel.replace_hints(...)

     to create the tuned kernel variant
   • Use

     ct.launch()

     for the actual kernel invocation

   • grid_fn

     correctly computes grid from config

   • args_fn

     passes all kernel arguments including tile sizes as

     ct.Constant[int]

   • hints_fn

     passes

     occupancy

     and/or

     num_ctas

     from config
   • VERIFY:

     exhaustive_search

     receives a

     list()

     of configs, not a raw generator.
7. (MANDATORY) Add DISABLE_AUTOTUNE support for CI and profiling: check

   os.environ.get("DISABLE_AUTOTUNE", "0") == "1"

   — when set, skip

   exhaustive_search

   entirely and fall back to

   ct.launch

   with the first valid config. This is required for:
   • CI determinism (autotune adds variable wall time)
   • NCU profiling (prevents autotune trial runs from cluttering the trace — see Pitfall #4)
   • Debugging (isolates kernel correctness from autotune behavior) Place the check before the cache lookup so that

     DISABLE_AUTOTUNE=1

     bypasses all autotune logic. Provide a hardcoded fallback config in case the generator yields zero configs.
   • VERIFY: Running with

     DISABLE_AUTOTUNE=1

     produces correct results and does not call

     exhaustive_search

     .
8. Test: Run correctness tests first (

   pytest -k "test_op and cutile"

   ), then benchmark.
   • VERIFY: Correctness passes with autotune enabled AND with

     DISABLE_AUTOTUNE=1

     .
9. Validate with A/B test: Compare autotune version vs fixed best-known config. See

   search-strategies.md

   for methodology.
   • VERIFY: Autotune version ≥ baseline (or within noise). If worse, check that the search space includes the original fixed config, and that

     replace_hints

     is being used correctly.
10. (MANDATORY) Run the test and verify performance before submitting.
    Execute the provided test script (e.g.

    ENABLE_TILE=1 python3 test.py

    ) and check:

    • correctness: PASS

    • speedup_over_fixed >= 1.0

      (autotuned must not be slower than fixed baseline)
    If

    speedup_over_fixed < 1.0

    :
    • Check that the search space includes the original fixed config (this guarantees no regression)
    • Check if

      replace_hints

      is being called on every code path — revisit Step 2 (if any path skips

      replace_hints

      , the decorator's fixed hints are used instead of autotuned values)
    • Expand search space if all configs perform similarly (see

      references/parameter-space-design.md

      → "Adapting Search Space")
    ⚠️ DO NOT submit without running the test at least once. Writing correct-looking code is not sufficient — autotuning bugs (silent hint override, split-buffer omission) are only caught at runtime.

torch.autograd.Function

• Place the tune-once/cache/launch logic in

  forward()

  only. The cached config is reused across calls.
• In

  backward()

  , using

  ct.launch

  with a fixed or cached config is often sufficient. However, if backward has its own independent search space (e.g. grouped GEMM dX and dW have separate optimal configs), autotuning is appropriate there too.
• Example:

  rope_embedding.py

  — forward uses

  exhaustive_search

  + cache with split-buffer, backward uses

  ct.launch

  with same-buffer (Q_in=Q_out).

1. Profile first: Use NCU (set

   DISABLE_AUTOTUNE=1

   ).
2. Expand (too narrow): add tile sizes,

   num_ctas

   (sm90+),

   swap_ab

   .
3. Prune (too slow): remove suboptimal configs, use arch-conditional yield, add size filters.
4. Re-validate: A/B test to confirm improvement.

• Keep total search space ≤ 30 configs — apply arch filters, tile size filters, and pruning rules until you're under the limit
• Use architecture-conditional yield to only generate relevant configs
• Prune the search space using architecture-conditional yield and size filters until total configs ≤ 30

exhaustive_search

ct.launch

• Search space generator functions

• exhaustive_search()

  calls and result handling

• kernel.replace_hints()

  for applying tuned hints
• Cache logic (key design, dict management)

• ct.launch()

  with tuned kernel

• DISABLE_AUTOTUNE

  fallback path

• Math flags (flush_to_zero, rounding_mode)
• Performance Hints (slice_hint, buffer_depth, copy_config)
• Memory access patterns (2D→1D gather/scatter conversion)
• Codegen optimizations (safe_offs → padding_value)
• Algorithm changes (K-loop split, load balancing)

• Math flags:

  flush_to_zero=True

  +

  rounding_mode=APPROX

  can provide 34-72% improvement for FMHA-class kernels (set via environment variables

  TILEIR_ENABLE_FTZ=1 TILEIR_ENABLE_APPROX=1

  or in kernel code). Causal chain: larger tiles initially decrease performance by 18-43% due to subnormal handling overhead; enabling FTZ+APPROX rescues this and flips the result to +34-72%. Math flags are therefore a prerequisite for large-tile configs to be effective on FMHA-class kernels.
• Performance Hints:

  slice_hint

  ,

  buffer_depth

  ,

  copy_config

  — requires modifying kernel IR code
• Memory access patterns: Using TMA loads (

  ct.load

  ) instead of

  ct.gather

  ; removing unnecessary bounds checks (

  check_bounds=False

  when safe)
• Codegen quality: Using

  padding_value

  parameter instead of manual

  ct.where

  masking; removing

  safe_offs

• Algorithm restructuring: K-loop split, load balancing, algebraic simplification