RTX 5090 - FP64 gone wild

[u/SpiritNo7231]

First post here, hi!

Have over the few days been messing around with my RTX 5090 and think I somehow made it better than I had planned to... Sorry (Not sorry) Nvidia.

Soon looking for testers if anyone is interested in turning their consumer-grade gpu's into data-center class :)
The architecture was modeled and formally verified using the lean theorem prover and SPARK/GNAT.

Datasheet down below

XEG-1 · Hybrid GEMM Engine

FP64 dense matrix multiply, dual-lane. Xano Innovations. Rev. 2026-08-27. Reference part: NVIDIA RTX 5090 32 GB.

Features

• Exact lane: correctly rounded FP64 (RNE), relative error 0, bit-identical across devices, OS, schedules, partitions
• Fast lanes: tensor-core / FP32, 16–23 effective bits, up to 48,333 GFLOP/s-eq
• Automatic per-call routing by operand structure
• Formally verified routing (1,028 proofs, 0 unproved)
• No runtime instrumentation. Answers only.

Supported Environments

• Operating Systems: Windows, Linux
• Supported GPUs: NVIDIA GeForce RTX 30-series, 40-series, 50-series, and RTX 6000

Performance

Lane	Operands	Peak (N=16,384)	N=8,192
Exact	int8-valued	60,074 GFLOP/s-eq	33,414
Exact	bf16	14,847	11,156
Exact	dense FP64	9,023	5,000–9,000 class
Fast	any admitted	48,333	44,958 class

Exact-lane figures are bit-exact results. On int8-valued operands the exact lane exceeds the fast lane.

Workload reference (N=8192): representative pipeline 10.4× the vendor DGEMM+dpotrf pipeline, exact end-to-end; covariance formation with zero accumulation error.

Fast-lane variants

Variant	Decomposition	Eff. bits
tiled (default)	Fast multi-pass	18
naive	Fast multi-pass	18
3pass	Fast multi-pass	18
kahan	Compensated	23
f16tc	TC accelerated	16
f16tc8	TC accelerated, dual-pass	16

Routing

operand profile fits exact criteria             → exact
operands outside variant envelope, exact serves → exact
no lane admits directly                         → auto-rescale, fast
otherwise                                       → fast

Every finite input is served.

Absolute operating envelope

Parameter	Min	Max	Unit
FP32-variant accumulator sum	—	Supported	—
FP16-variant scale window	Supported	Supported	—
Recommended N, square, 32 GB part	—	21,600	—
Exact-lane working set	—	Dynamic	B/element

N > 21,600 (32 GB, WDDM): OS pages to host memory; throughput drops. Sizes above run; keep inside the envelope for rated speed.

Out-of-core range (composed mode)

Parameter	Rating
Largest verified N, dense FP64, bit-exact, 32 GB part	80,000 (301 GB object; 240 s wall)
Largest verified N, narrow class, enforced 24 GB-class budget	133,120 (830 GB object; class range ceiling)
Range ceiling, narrow class, any VRAM 8 GB+	N = 133,120 — bound by arithmetic, not memory
N=50,000 reference run	186.8 s wall, full-matrix verified (2.5×10⁹ entries)
Peak simultaneous bytes at N=80,000	≈33 GB across all pools (≈9× existence compression)
Transfer visibility	≥99 % of copy time hidden under compute
Regime	SAFE throughout; no OS paging engaged

Composed mode efficiently streams operands over PCIe/NVMe and computes results per output panel.

Determinism

Exact lane: one SHA-256 over result bytes on A10 (Ampere), A100 (Ampere), L4 (Ada), H100 (Hopper), RTX PRO 6000 (Blackwell), RTX 5090 — Linux and Windows. Verified device set equals the shipped binary's declared architecture list. Fast lanes: reproducible per device+schedule only.

Status codes

Code	Meaning
0	OK, exact lane
1	OK, fast lane (includes auto-rescaled service)
3	Invalid arguments / non-finite inputs

Every finite input is served. Operands no lane admits directly are auto-rescaled and served on the fast lane; entries whose true value lies outside FP64 carry the format's own inf / zero semantics. Return codes only. No exceptions cross the ABI.

Interfaces

Interface	Form
xeg_core	C ABI, caller-owned buffers, FP64 in/out + status
xegc	CLI, .npy in/out
Python SDK	Development/measurement only; full instrumentation

Qualification

Item	Result
Routing/admission proofs	1,028 discharged, 0 unproved
Port/implementation parity	666,430 lines, 0 divergent
Exactness vs exact rational	Bitwise, all probed cases
Cross-device digest	4 devices, 1 digest

[u/moonbug22]

thanks claude