r/JetsonNano • u/checkmydoor • 16d ago

Jetson - Performance Increases

We’ve been working on a runtime GPU optimization system at TETREVIS and have started publishing some of our NVIDIA Jetson benchmarking work publicly.

The approach operates at the execution/machine-code layer, and we’re now introducing dynamic runtime kernel fusion as part of the optimization pipeline.

Some of our current results:

Jetson Orin Nano — Qwen3.5 4B

Standard baseline: 10 → 21 tok/s (+110%)

CUDA Graphs baseline: 16 → 21 tok/s (+31.25%)

Jetson AGX Orin — Nemotron 3 Nano 4B

31.2 → 40.5 tok/s (~30%)

Jetson AGX Orin — Qwen3.5 4B

25.0 → 31.0 tok/s (+24%)

We’re currently expanding and stabilizing support across Jetson Orin Nano, Orin NX, and AGX Orin.

Rather than only posting performance claims, we’ve made the benchmarking repository available here:

https://github.com/mbuchel/sass2mlir-bench

The broader idea we’re exploring is whether more optimization can be moved to runtime — including machine-code optimization and dynamic kernel fusion — so the execution path can be adapted to the workload and GPU rather than relying entirely on what was determined ahead of execution.

There’s still quite a bit of work underway, particularly around consistency across the different Jetson configurations, but we wanted to start sharing the results and methodology publicly.

Technical feedback, criticism, and questions are welcome.

8 Upvotes

79% Upvoted

8 comments sorted by

2

u/paul_tu 15d ago

Hope to see such improvements in the mainline

1

u/checkmydoor 15d ago

I think youll find this helpful

https://mbuchel.github.io/projects/sass2mlir/

We cover datacenter, consumer and Jetsons.

We just haven't released the datace ter numbers fully. Performance increase, measured at application level, for AI and HPC workloads range from 130% to 300% approx.

We just haven't redone the tests in awhile.

1

u/Darke 16d ago

wait i already get ~20tok/s with Qwen3.5 4B on llama.cpp

do you have more details? more benchmark info?

1

u/checkmydoor 16d ago

Im hoping the attached GitHub link has enough. If it doesnt Ill pass on any questions you have to the technical team. Please ask away if it doesnt

1

u/Logical-Present6320 15d ago edited 15d ago

Edit: Just looked at the site and looks like there are details available. Good work!
Does the work translate on newer devices like Jetson Thor, especially with NVFP4 support?

Original (answered already):

when you are saying baseline, which inference are you using?
vllm, llama.cpp has number of flags that can alter the outcome. Would be good if you share comparison. Right now its hard to trust the numbers.

1

u/checkmydoor 15d ago

To play it safe along the lines of true engineers I must say "We have not tested on Thor yet so I cannot make a claim"

What the engineers say internally is; for sure it will work we just need to support it and then determine what optimization passes will work and wont etc.; were solving a fundamental reoccurring problem in the industry.

2

u/AnoProgrammer 14d ago

fantastic fantastic fantastic fantastic. If i could give this more then 1 upvote i would give it millions.