Motivation

Several CUDA linalg paths use cuSOLVER by looping over independent batch items on the current CUDA stream. This is correct, but it can underutilize the GPU for medium-sized matrices when there is no suitable batched cuSOLVER API, or when PyTorch's heuristic selects a looped cuSOLVER path.

The recent LU investigation showed that these independent calls can overlap well when issued onto multiple nonblocking CUDA streams. This issue is not just for getrf; the desired outcome is to investigate and improve looped cuSOLVER batch execution across CUDA linalg where auxiliary-stream execution makes sense.

Local evidence

Standalone C++ benchmark on:

• GPU: NVIDIA RTX 6000 Ada Generation
• CUDA runtime: 13.2
• cuSOLVER reported version: 12200
• dtype: float32
• operation: independent cusolverDnSgetrf calls over a batch

In this standalone probe, changing the stream on one cuSOLVER handle before each call performed about the same as using separate handles. That still needs validation against cuSOLVER's intended handle/stream usage and PyTorch's handle pool assumptions.

This does not replace the need for a batch-aware cuBLAS/cuSOLVER heuristic. For example, in the same investigation cublasSgetrfBatched for M=512, batch=64 was still around 6.3 ms, much faster than the multistream looped cuSOLVER result. The multistream approach looks more relevant once PyTorch has already selected a looped cuSOLVER path, especially for larger matrices or operations without a fast batched alternative.

Scope

Examples in aten/src/ATen/native/cuda/linalg/BatchLinearAlgebraLib.cpp include:

• lu_factor_looped_cusolver / getrf
• lu_solve_looped_cusolver / getrs
• apply_cholesky_cusolver_potrf_looped / potrf and xpotrf
• QR paths: geqrf, orgqr, ormqr
• SVD fallback paths: gesvd, gesvdj where implemented as per-batch loops
• Hermitian/general eigensolver paths: syevd, xsyevd, xgeev
• LDL paths that are currently dispatched as looped cuSOLVER

Some of these paths also allocate workspace inside the batch loop. Workspace lifetime and reuse matter for any auxiliary-stream version because concurrent calls cannot share the same workspace unless their lifetimes do not overlap.

Constraints

Any implementation should preserve normal PyTorch stream semantics and existing behavior:

• Auxiliary streams should wait on the current stream before reading inputs or outputs.
• The current stream should wait on auxiliary streams before the op returns to the user.
• Inputs, outputs, pivots, infos, and workspaces used on auxiliary streams need correct caching allocator stream recording so storage is not reused while still in use.
• Operations that already have efficient batched cuSOLVER/cuBLAS paths should keep using those when faster.
• Error reporting through info tensors and check_errors behavior must remain unchanged.

Related:

• #181997
• #181998
• #181999

cc @jerryzh168 @ptrblck @msaroufim @eqy @tinglvv @nWEIdia @jianyuh @nikitaved @mruberry @walterddr @xwang233 @Lezcano