How to Use CuPy linalg for GPU Linear Algebra: A Practical Guide to cuSOLVER and cuBLAS
CuPy's cupy.linalg module mirrors the NumPy linear algebra API while offloading every operation to NVIDIA cuSOLVER and cuBLAS, letting you solve linear systems, invert matrices, and compute decompositions directly on GPU-resident cupy.ndarray objects.
The cupy/cupy repository provides a near-complete GPU replacement for numpy.linalg through its cupy.linalg and cupyx.linalg namespaces. These modules route your array data to highly optimized CUDA kernels, so you can accelerate dense linear algebra without rewriting NumPy code. Whether you need a simple linear solve or a batched matrix inverse, CuPy linalg keeps both the syntax and the performance on your side.
Architecture of CuPy linalg
CuPy linalg is organized into layered components that separate the user-facing API from the underlying CUDA libraries. According to the cupy/cupy source code, the public functions in cupy.linalg are thin wrappers that validate input, select a compute dtype, and then dispatch to cuSOLVER or cuBLAS.
- Public API layer. Functions such as
cp.linalg.solve,cp.linalg.inv, andcp.linalg.lstsqlive incupy/linalg/_solve.pyand provide the same signatures as NumPy. The top-level entry point iscupy/linalg/__init__.py. - Backend bindings. The actual GPU work is performed by routines imported from
cupyx.lapackandcupy.cuda.cublas, which wrap cuSOLVER kernels likegesvand cuBLAS batched routines. - Utility layer. The file
cupy/linalg/_util.py(lines 18–59) contains_assert_cupy_array,_assert_2d, and_assert_stacked_square, plus_util.linalg_common_typefor dtype selection. It also respects the optionalCUPY_DEFAULT_PRECISIONenvironment variable when choosing float32 versus float64 math.
After each kernel launch, CuPy checks for failures via _check_cusolver_dev_info_if_synchronization_allowed. If the CUDA routine reports an error, the helper raises a numpy.linalg.LinAlgError unless the user has silenced errors with cupyx.seterr(linalg='ignore').
Solving Linear Systems with CuPy linalg
Dense System Solver (solve)
The workhorse for dense linear systems is cp.linalg.solve. As implemented in cupy/linalg/_solve.py (lines 13–85), it first validates dimensions with _util helpers, then chooses between a batched cuBLAS path and a single-matrix cuSOLVER path.
import cupy as cp
A = cp.array([[3., 2.], [1., 4.]])
b = cp.array([1., 3.])
x = cp.linalg.solve(A, b) # cp.array([0.2, 0.7])
For batched inputs—tensors with shape (..., N, N)—CuPy automatically reshapes the batch dimension. When the inner matrix size is small enough (a.shape[-1] <= get_batched_gesv_limit()), it calls cupy.cublas.batched_gesv; otherwise it falls back to a Python loop over lapack.gesv.
Cholesky Solver (cho_solve)
If you already hold a Cholesky factorization, cupyx.linalg.cho_solve avoids redundant factorization. The implementation in cupyx/linalg/_solve.py (lines 34–96) forwards to cupyx.lapack.potrs after an optional copy for C-contiguity.
A = cp.array([[4., 1.], [1., 3.]])
c = cp.linalg.cholesky(A) # lower-triangular factor
b = cp.array([1., 2.])
x = cp.linalg.cho_solve((c, True), b) # True → lower storage
Matrix Inversion and Pseudoinverse in CuPy linalg
Hermitian Inverse (invh)
For Hermitian (or real symmetric) positive-definite matrices, cupyx.linalg.invh is more efficient than a generic inverse. Defined in cupyx/linalg/_solve.py (lines 9–31), it constructs an identity matrix and calls lapack.posv to solve for the inverse in place.
H = cp.array([[4., 1. + 0j], [1. - 0j, 3.]])
H_inv = cp.linalg.invh(H) # uses Cholesky internally
Standard and Batched Inverse (inv)
cp.linalg.inv in cupy/linalg/_solve.py (lines 58–77) handles both single and stacked matrices. When the input has three or more dimensions, the internal _batched_inv routine selects the cuBLAS batched LU kernel (sgetrfBatched, dgetrfBatched, etc.) based on dtype, performs the LU factorization, and completes the inversion.
A = cp.array([[3., 2.], [1., 4.]])
A_batch = cp.stack([A, A * 2]) # shape (2, 2, 2)
A_inv = cp.linalg.inv(A_batch) # shape (2, 2, 2)
Moore-Penrose Pseudoinverse (pinv)
cp.linalg.pinv follows the classic SVD-based definition found in NumPy. The CuPy linalg implementation in cupy/linalg/_solve.py (lines 19–63) computes the singular value decomposition, discards values below rcond * max(s), and recombines the factors.
A = cp.random.randn(4, 2)
A_pinv = cp.linalg.pinv(A, rcond=1e-12)
Least-Squares and Tensor Operations in CuPy linalg
Over-Determined Least-Squares (lstsq)
cp.linalg.lstsq solves over-determined systems via an SVD-based pseudo-inverse algorithm. The source code in cupy/linalg/_solve.py (lines 34–122) mirrors the NumPy flow: decompose with cupy.linalg.svd, filter small singular values, and compute the solution vector.
A = cp.random.randn(8, 3)
b = cp.random.randn(8)
x, residuals, rank, s = cp.linalg.lstsq(A, b)
Tensor Inverse (tensorinv)
For higher-dimensional arrays, cp.linalg.tensorinv treats specified axes as a single matrix block. As shown in cupy/linalg/_solve.py (lines 66–103), it flattens the tensor, calls cp.linalg.inv, and restores the original block structure.
T = cp.arange(24).reshape(2, 2, 2, 3)
T_inv = cp.linalg.tensorinv(T, ind=2)
Sparse Linear Algebra Extensions
Beyond dense arrays, cupyx.linalg.sparse re-exports SciPy-compatible sparse solvers built on cuSPARSE and cuSOLVER. These live under cupyx/scipy/sparse/linalg/ and allow GPU acceleration for CSR and CSC matrices when dense storage is impractical.
Summary
cupy.linalgandcupyx.linalgexpose a NumPy-compatible GPU linear algebra API backed by cuSOLVER and cuBLAS.- Core solvers like
solve,lstsq, andinvlive incupy/linalg/_solve.py, while Hermitian and Cholesky helpers live incupyx/linalg/_solve.py. - The
cupy/linalg/_util.pylayer validates shapes, ensures inputs arecupy.ndarrays, and selects compute dtypes via_util.linalg_common_type. - Batched operations are automatic: small matrices use
cupy.cublas.batched_gesv, and larger batches fall back to loops overlapack.gesv. - Error handling checks
devInfoafter CUDA calls and raisesnumpy.linalg.LinAlgErrorunless configured otherwise withcupyx.seterr.
Frequently Asked Questions
Is CuPy linalg a direct replacement for numpy.linalg?
Yes, cupy.linalg is intentionally designed to mirror the NumPy API, so functions like solve, inv, and lstsq accept nearly identical arguments and return cupy.ndarray objects. The main difference is that inputs must already reside on the GPU as cupy.ndarrays, whereas NumPy operates on host memory.
How does CuPy linalg process multiple matrices at once?
CuPy linalg automatically detects batched inputs when arrays have more than two dimensions. For small inner dimensions it routes to cupy.cublas.batched_gesv, and for larger batches it iterates via lapack.gesv inside a Python loop, all without requiring explicit user flags.
Which CUDA libraries does CuPy linalg call under the hood?
CuPy linalg delegates its heavy computation to NVIDIA cuSOLVER and cuBLAS, with optional sparse support through cuSPARSE. You can see these calls inside cupy/linalg/_solve.py and cupyx/linalg/_solve.py where functions import cupyx.lapack and cupy.cuda.cublas.
Can I change how CuPy linalg reacts to numerical errors?
By default, CuPy inspects devInfo after each kernel and raises numpy.linalg.LinAlgError when the CUDA routine reports failure. You can override this behavior by calling cupyx.seterr(linalg='ignore') or using cupyx.seterr(linalg='raise') for stricter enforcement.
Have a question about this repo?
These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:
curl -s "https://instagit.com/install.md" Maintain an open-source project? Get it listed too →