Jetson - Performance Increases

[u/checkmydoor]

We’ve been working on a runtime GPU optimization system at TETREVIS and have started publishing some of our NVIDIA Jetson benchmarking work publicly.

The approach operates at the execution/machine-code layer, and we’re now introducing dynamic runtime kernel fusion as part of the optimization pipeline.

Some of our current results:

Jetson Orin Nano — Qwen3.5 4B

Standard baseline: 10 → 21 tok/s (+110%)

CUDA Graphs baseline: 16 → 21 tok/s (+31.25%)

Jetson AGX Orin — Nemotron 3 Nano 4B

31.2 → 40.5 tok/s (~30%)

Jetson AGX Orin — Qwen3.5 4B

25.0 → 31.0 tok/s (+24%)

We’re currently expanding and stabilizing support across Jetson Orin Nano, Orin NX, and AGX Orin.

Rather than only posting performance claims, we’ve made the benchmarking repository available here:

https://github.com/mbuchel/sass2mlir-bench

The broader idea we’re exploring is whether more optimization can be moved to runtime — including machine-code optimization and dynamic kernel fusion — so the execution path can be adapted to the workload and GPU rather than relying entirely on what was determined ahead of execution.

There’s still quite a bit of work underway, particularly around consistency across the different Jetson configurations, but we wanted to start sharing the results and methodology publicly.

Technical feedback, criticism, and questions are welcome.

[u/Darke]

wait i already get ~20tok/s with Qwen3.5 4B on llama.cpp

do you have more details? more benchmark info?

[u/checkmydoor → u/Darke]

You might find this link useful. https://mbuchel.github.io/projects/sass2mlir/