vllm - 💡(How to fix) Fix [Hybrid SSM] Investigate accuracy divergence between `mamba_chunk_scan` and `selective_state_update` kernels

Context

PR #43186 lowered the gsm8k accuracy threshold for ibm-granite/granite-4.0-h-tiny under NIXL PD disagg after #42430 rerouted the D-side 1-token recompute from mamba_chunk_scan_combined_varlen to selective_state_update (SSU).

The two kernels produce different bf16 outputs due (supposedly) different reduction orders:

• mamba_chunk_scan_combined_varlen goes through bf16 matmul with intermediate bf16 casts (tl.dot)
• selective_state_update keeps the SSM step in fp32 throughout

As noted in this comment, this seems like a large accuracy drop for something meant to be algebraically equivalent.

Goal

Investigate the kernel-level numerical divergence in isolation to:

1. Quantify the magnitude: Run both kernels on identical inputs for a single prompt (prefill + decode) and measure the output divergence (max abs diff, relative error distribution).
2. Determine hardware dependence: Test on at least Ampere (A100) and Hopper (H100) to see if the divergence magnitude or direction is architecture-dependent.
3. Recommend a fix or acceptable tolerance: Based on findings, either align the kernels' numeric behavior or document the expected tolerance.

Reproduction Plan

• Isolate the two kernels (mamba_chunk_scan_combined_varlen and selective_state_update) with a single-prompt workload (1 prefill + N decode steps).
• Compare outputs element-wise against each other and against an fp64 reference.
• Run on Ampere (A100) and Hopper (H100) hardware.

References

• PR that lowered threshold: #43186
• PR that rerouted to SSU: #42430