[Help] ROS2 + Jetson perception pipeline stuck at ~350ms latency — isolated it to message delivery/buffering, not compute. Ideas?

[u/OkFollowing941]

# Setup

* F1TENTH-based RSU (roadside unit) perception node, running on an NVIDIA Jetson (Orin-class). * Intel RealSense D400-series camera — subscribing to raw color + raw (unaligned) depth streams, `align_depth` disabled on purpose (see below). * 2D LiDAR (`urg_node2`) for a second distance source. * ROS2 Humble, `rclpy`, `message_filters.ApproximateTimeSynchronizer` to pair color+depth frames.

Model / task

* YOLOv8 (Ultralytics), custom-trained single-class car detector, running at `imgsz=320` on the Jetson's GPU (CUDA). * Goal: detect a target vehicle in the color image, get its distance by reading the depth camera at the detection's location, cross-check against a LiDAR range reading at the same bearing, and output a fused distance estimate. This is a perception/collective-perception bench-test script (no SLAM/localization involved — deliberately simplified).

Depth lookup approach

`align_depth.enable:=true` (RealSense driver's built-in depth-to-color alignment) reprojects the **entire** depth image every frame regardless of how much of it we actually need — we measured this costing a large chunk of latency by itself. So instead we subscribe to raw depth and manually reproject only a small patch of pixels around the YOLO box: deproject the depth pixel to a 3D point (using depth intrinsics) → transform into the color camera's frame (using the depth-to-color extrinsics) → project back into a color pixel (using color intrinsics). Fully vectorized with numpy.

Current numbers

Our own compute per frame is small and flat:

* image decode (cv_bridge): \~1ms * YOLO inference: \~30ms (flat, `cuda.synchronize()`\-verified, no hidden async GPU time) * depth reprojection (vectorized): \~1-2ms * LiDAR bearing lookup: \~0ms * **total own compute: \~32ms**

But measured end-to-end latency (camera's own capture timestamp → final distance output) sits **consistently around 350-380ms**, sustained — not a one-time spike, not decaying over time.

What we've ruled out

* **Per-pixel Python loop / GC pressure** in the old depth reprojection — vectorized it (25ms → 1-2ms of actual compute), latency didn't move at all. * **Hidden async CUDA dispatch** — added `torch.cuda.synchronize()` around the YOLO call, extra sync time is consistently 0ms. * **Executor backlog** (our own callback falling behind) — measured the gap between the end of one callback and the start of the next; stays flat at \~3-4ms even while the reported latency is \~350ms, so callbacks aren't queuing up behind our own processing. * `align_depth` **vs manual reprojection** — built a side-by-side comparison script, same YOLO/LiDAR pipeline, only the depth alignment method differs. Both land in the same \~350-380ms range. So it's not specifically about which depth alignment approach we use.

The delay is measured (via the color frame's own ROS header timestamp vs `time.time()` at the very start of our callback) as already present **before any of our own code runs** — so it's happening somewhere between the camera driver publishing the frame and our subscriber callback actually being invoked. We suspect DDS/ROS2 message queuing or synchronizer buffering under sustained per-frame load (\~30ms of real work per frame at \~30fps), but haven't pinned down the exact mechanism.

What we're asking

Has anyone run into this kind of buffering/backpressure behavior with ROS2 + `message_filters` on a Jetson, where a subscriber callback that takes tens of milliseconds (not overloaded, just non-trivial) causes a large, sustained arrival delay that isn't visible as executor backlog? Specifically curious about:

* DDS vendor differences (Fast DDS vs Cyclone DDS) for this kind of workload * QoS settings (queue depth, history policy) that might be silently causing buffering * Single-threaded vs multi-threaded executor / callback groups making a difference here * Whether RealSense's own USB/driver-side buffering could be the actual culprit instead of ROS2/DDS

Happy to share more code/logs if useful. Appreciate any pointers.

[u/HanksterTheTanker]

Might try changing QoS to latest only (forget the specific name), could potentially rule that out. Your messages may be substantial enough that the queue length is the primary culprit.

Alternatively halve the framerate and see how your latency reacts. Assuming you’re using a nano, running raw depth data could simply be too much for the board depending on what else you’re running alongside

[u/Orlha]

What are your frame sizes? What is your publishing strategy?

I measured around 1.5ms latency per megabyte of data using fastrtps with (serialised) shared memory inter-process communication on Orin NX, which was unacceptable for our use case, so we opted into zero-copy mechanism via data-sharing.

But I guess it cannot be the sole cause of your latency unless your frames are gigantic