Operator MFU Calculator

Basic Concepts

• MFU Definition
  MFU (Machine FLOP Utilization) is defined as: $$ \text{MFU} = \frac{\text{Actual FLOPs Generated by Computation}}{\text{Theoretical FLOPs Executable by Hardware in the Same Time}} = \frac{\text{Achieved FLOPs}}{\text{Peak FLOPs}} $$
• Unit Conventions
  • FLOPs: Number of floating-point operations
  • TFLOPs/s: Trillions of floating-point operations per second
  • Pay attention to unit unification during calculation, for example:
    • Actual FLOPs / Execution Time = Achieved FLOPs/s
    • Achieved TFLOPs/s = Achieved FLOPs/s ÷ 1e12

Reference for Theoretical Peak Computing Power (Peak FLOPs) of Common Chips

• Huawei Ascend 910B1
  • FP16/BF16：≈ 378.88 TFLOPs/s
• Huawei Ascend 910B2
  • FP16/BF16：≈ 353.89 TFLOPs/s
• Huawei Ascend 910B3
  • FP16/BF16：≈ 294.91 TFLOPs/s
• Huawei Ascend 910B4
  • FP16/BF16：≈ 270 TFLOPs/s

1. First ask for the specific model, precision mode (FP32/FP16/BF16/FP8, etc.), and whether Tensor Core / Matrix Core is used.
2. If the user only provides a general model, clearly state that the typical approximate values from the above table are used, and remind the user that the result is a rough estimate.
3. It is recommended that users refer to the peak computing power given in official documents and supplier reports first to obtain a more accurate MFU.

Matmul / GEMM FLOPs Calculation

• Standard Matrix Multiplication (GEMM)
  For matrix multiplication of shapes $(M, K)$ and $(K, N)$: $$ \text{FLOPs} \approx 2 \times M \times N \times K $$
  • The factor of 2 comes from "one multiplication + one addition".
• Matmul with Batch Dimension
  For batched matmul of shapes $(B, M, K)$ and $(B, K, N)$: $$ \text{FLOPs} \approx 2 \times B \times M \times N \times K $$
• Examples of Common Scenarios (can be directly analogized)
  • Linear layer: Input $(B, L, D_\text{in})$, Weight $(D_\text{in}, D_\text{out})$
    → Can be regarded as $M = B \times L,\ K = D_\text{in},\ N = D_\text{out}$.
  • $QK^T$ in Attention: $Q=(B, H, L_q, D_h),\ K=(B, H, L_k, D_h)$
    → Can be regarded as $B' = B \times H,\ M = L_q,\ N = L_k,\ K = D_h$.

FlashAttention FLOPs Calculation

Input Layout Description

• BNSD: $(B, N, S, D)$ → Use directly
• BSND: $(B, S, N, D)$ → Convert to $(B, N, S, D)$
• BSH: $(B, S, D)$ → Convert to $(B, 1, S, D)$ (single head)
• SBH: $(S, B, D)$ → Convert to $(B, 1, S, D)$ (single head)
• TND: $(T, N, D)$ → Varlen scenario, special handling is required, actual sequence length information is needed

TND Layout Formula

input_layout == "TND"

actual_seq_qlen

actual_seq_kvlen

1. Parse Actual Sequence Lengths
   Convert from cumulative lengths to actual lengths for each sample: $$ \text{q_lens} = [\text{actual_seq_qlen}[0], \text{actual_seq_qlen}[1] - \text{actual_seq_qlen}[0], \text{actual_seq_qlen}[2] - \text{actual_seq_qlen}[1], \ldots] $$ $$ \text{kv_lens} = [\text{actual_seq_kvlen}[0], \text{actual_seq_kvlen}[1] - \text{actual_seq_kvlen}[0], \text{actual_seq_kvlen}[2] - \text{actual_seq_kvlen}[1],\ldots] $$ (Remove trailing zeros, keep only valid lengths)
2. Calculate Sequence Workload
   $$ \text{acl_seq_workload} = \sum_{i} \text{q_lens}[i] \times \text{kv_lens}[i] $$
3. Calculate FLOPs
   Let the shape of $Q$ be $(T_q, N, D_q)$, and the shape of $K$ be $(T_k, N, D_k)$: $$ \text{FLOPs} = 2 \times N \times (D_q + D_k) \times \text{acl_seq_workload} $$

Common Layout Formula (BNSD/BSND/BSH/SBH)

input_layout

sparse_mode

1. Unified Dimension Representation
   Convert input to $(B, N, S, D)$ format:
   • $Q$: $(q_b, q_n, q_s, q_d)$
   • $K$: $(k_b, k_n, k_s, k_d)$
2. Basic Full Attention FLOPs
   $$ \text{full_attention} = 2 \times q_b \times q_n \times q_s \times k_s \times (q_d + k_d) $$
3. Adjust According to sparse_mode
   • sparse_mode == 0 (full attention):
     $$ \text{FLOPs} = \text{full_attention} $$
   • sparse_mode == 2 or 3, and $q_s == k_s$ (causal or similar, equal sequence lengths):
     $$ \text{FLOPs} = \text{full_attention} \times 0.5 $$
   • sparse_mode == 2, and $q_s > k_s$ (causal, longer query):
     $$ \text{FLOPs} = \text{full_attention} \times \frac{q_s \times k_s - k_s \times k_s / 2}{k_s \times k_s} $$
   • sparse_mode == 3, and $q_d > k_d$ (special sparse mode):
     $$ \text{FLOPs} = \text{full_attention} \times \frac{k_s \times k_s / 2}{q_s \times k_s} $$
   • sparse_mode == 2, and $q_d < k_d$:
     $$ \text{FLOPs} = \text{full_attention} \times \frac{q_s \times q_s / 2}{q_s \times k_s} $$
   • sparse_mode == 3, and $q_d < k_d$:
     $$ \text{FLOPs} = \text{full_attention} \times \frac{q_s \times k_s - q_s \times q_s / 2}{q_s \times k_s} $$

FlashAttention Calculation Notes

• Required Information:
  • Input layout (input_layout): TND or BNSD/BSND/BSH/SBH
  • For TND: Need

    actual_seq_qlen

    and

    actual_seq_kvlen

    (cumulative length arrays)
  • For Common layout: Need

    sparse_mode

    (0/2/3)
  • Input tensor shapes (input_shapes)
• Common sparse_mode Meanings:

  • 0

    : Full attention (no sparsity)

  • 2

    : Usually represents causal attention (causal mask)

  • 3

    : Other sparse modes
• If Key Parameters Are Missing (such as sparse_mode or actual_seq_qlen), clearly inform the user that these information need to be obtained from

  operator_args

  .

Standard Steps for Calculating MFU

1. Confirm Sufficient Information
   Ask the user for the following information (clearly request if missing):
   • Operator type (e.g., matmul / GEMM / FlashAttention, etc.).
   • Dimensions of tensors involved in computation (including key dimensions such as batch / head / sequence, etc.).
   • Runtime of a single operator execution (e.g., milliseconds ms).
   • Theoretical peak computing power of a single hardware card (e.g., 312 TFLOPs/s, specify whether it is FP16/BF16 or FP8, etc.).
2. Calculate Operator FLOPs
   • Calculate the FLOPs of a single call using the above formulas based on the operator type and dimensions.
   • If the user provides "how many times this operator is included per iteration" or "multiple identical operators", calculate the single call first, then multiply by the number of calls.
3. Calculate Achieved FLOPs/s
   • First convert the runtime to seconds, for example: $t_\text{s} = \text{time_ms} / 1000$.
   • Achieved FLOPs/s = FLOPs / $t_\text{s}$.
   • Then convert to TFLOPs/s: Achieved TFLOPs/s = Achieved FLOPs/s ÷ 1e12.
4. Calculate MFU
   • MFU = Achieved TFLOPs/s ÷ Peak TFLOPs/s.
   • Finally present it as a percentage, e.g., 0.42 → 42%.
5. Explain the Result
   • Briefly explain what this MFU represents, for example:
     • Below 20%: Usually the operator is far from utilizing full computing power, which may be affected by memory bandwidth, launch overhead, irregular shape, etc.
     • 30%–60%: Above-average level, many general workloads are roughly in this range.
     • Above 70%: The operator shape, parallelism, and implementation are relatively close to the device's upper limit.

Answer Format Requirements

1. When calculating MFU according to the steps provided by this Skill, clearly state at the beginning of the answer: "(This answer is based on the MFU calculation specifications of the op-mfu-calculator Skill)"
2. First repeat the input information (including operator type, tensor dimensions, time, peak computing power).
3. List key formulas (FLOPs, Achieved TFLOPs/s, MFU), and substitute specific numbers to show the intermediate calculation process.
4. Provide the final MFU value (retain 2–3 significant figures, in percentage form).
5. Briefly analyze possible reasons for this MFU or optimization directions (e.g., too small batch, too small K dimension, memory bandwidth bottleneck, etc.).