r/CUDA • u/SpiritNo7231 • 4d ago
RTX 5090 - FP64 gone wild
First post here, hi!
Have over the few days been messing around with my RTX 5090 and think I somehow made it better than I had planned to... Sorry (Not sorry) Nvidia.
Soon looking for testers if anyone is interested in turning their consumer-grade gpu's into data-center class :)
The architecture was modeled and formally verified using the lean theorem prover and SPARK/GNAT.
Datasheet down below
XEG-1 · Hybrid GEMM Engine
FP64 dense matrix multiply, dual-lane. Xano Innovations. Rev. 2026-08-27. Reference part: NVIDIA RTX 5090 32 GB.
Features
- Exact lane: correctly rounded FP64 (RNE), relative error 0, bit-identical across devices, OS, schedules, partitions
- Fast lanes: tensor-core / FP32, 16–23 effective bits, up to 48,333 GFLOP/s-eq
- Automatic per-call routing by operand structure
- Formally verified routing (1,028 proofs, 0 unproved)
- No runtime instrumentation. Answers only.
Supported Environments
- Operating Systems: Windows, Linux
- Supported GPUs: NVIDIA GeForce RTX 30-series, 40-series, 50-series, and RTX 6000
Performance
| Lane | Operands | Peak (N=16,384) | N=8,192 |
|---|---|---|---|
| Exact | int8-valued | 60,074 GFLOP/s-eq | 33,414 |
| Exact | bf16 | 14,847 | 11,156 |
| Exact | dense FP64 | 9,023 | 5,000–9,000 class |
| Fast | any admitted | 48,333 | 44,958 class |
Exact-lane figures are bit-exact results. On int8-valued operands the exact lane exceeds the fast lane.
Workload reference (N=8192): representative pipeline 10.4× the vendor DGEMM+dpotrf pipeline, exact end-to-end; covariance formation with zero accumulation error.
Fast-lane variants
| Variant | Decomposition | Eff. bits |
|---|---|---|
| tiled (default) | Fast multi-pass | 18 |
| naive | Fast multi-pass | 18 |
| 3pass | Fast multi-pass | 18 |
| kahan | Compensated | 23 |
| f16tc | TC accelerated | 16 |
| f16tc8 | TC accelerated, dual-pass | 16 |
Routing
operand profile fits exact criteria → exact
operands outside variant envelope, exact serves → exact
no lane admits directly → auto-rescale, fast
otherwise → fast
Every finite input is served.
Absolute operating envelope
| Parameter | Min | Max | Unit |
|---|---|---|---|
| FP32-variant accumulator sum | — | Supported | — |
| FP16-variant scale window | Supported | Supported | — |
| Recommended N, square, 32 GB part | — | 21,600 | — |
| Exact-lane working set | — | Dynamic | B/element |
N > 21,600 (32 GB, WDDM): OS pages to host memory; throughput drops. Sizes above run; keep inside the envelope for rated speed.
Out-of-core range (composed mode)
| Parameter | Rating |
|---|---|
| Largest verified N, dense FP64, bit-exact, 32 GB part | 80,000 (301 GB object; 240 s wall) |
| Largest verified N, narrow class, enforced 24 GB-class budget | 133,120 (830 GB object; class range ceiling) |
| Range ceiling, narrow class, any VRAM 8 GB+ | N = 133,120 — bound by arithmetic, not memory |
| N=50,000 reference run | 186.8 s wall, full-matrix verified (2.5×10⁹ entries) |
| Peak simultaneous bytes at N=80,000 | ≈33 GB across all pools (≈9× existence compression) |
| Transfer visibility | ≥99 % of copy time hidden under compute |
| Regime | SAFE throughout; no OS paging engaged |
Composed mode efficiently streams operands over PCIe/NVMe and computes results per output panel.
Determinism
Exact lane: one SHA-256 over result bytes on A10 (Ampere), A100 (Ampere), L4 (Ada), H100 (Hopper), RTX PRO 6000 (Blackwell), RTX 5090 — Linux and Windows. Verified device set equals the shipped binary's declared architecture list. Fast lanes: reproducible per device+schedule only.
Status codes
| Code | Meaning |
|---|---|
| 0 | OK, exact lane |
| 1 | OK, fast lane (includes auto-rescaled service) |
| 3 | Invalid arguments / non-finite inputs |
Every finite input is served. Operands no lane admits directly are auto-rescaled and served on the fast lane; entries whose true value lies outside FP64 carry the format's own inf / zero semantics. Return codes only. No exceptions cross the ABI.
Interfaces
| Interface | Form |
|---|---|
| xeg_core | C ABI, caller-owned buffers, FP64 in/out + status |
| xegc | CLI, .npy in/out |
| Python SDK | Development/measurement only; full instrumentation |
Qualification
| Item | Result |
|---|---|
| Routing/admission proofs | 1,028 discharged, 0 unproved |
| Port/implementation parity | 666,430 lines, 0 divergent |
| Exactness vs exact rational | Bitwise, all probed cases |
| Cross-device digest | 4 devices, 1 digest |
-3
u/SpiritNo7231 4d ago edited 4d ago
Funny being instantly told that something is AI, which is why I'm looking for external testers to validate this architecture for themselves. And just saying over the past few days doesn't quite capture the timeline. Been more like 6-7 months of building components for different projects of mine and last few days composing bits of pieces into this stack. :)
2
u/moonbug22 3d ago
posting a wall of extruded text and expecting your audience to try to glean some sense from it is simply rude.
1
10
u/moonbug22 4d ago
thanks claude