RTX Lidar point cloud becomes geometrically inconsistent during rapid sensor motion

[yuheisugano]

Hi Isaac Sim team,
I am observing the following issue with RTX lidar simulation in a moving-vehicle setup. I would appreciate any guidance or suggestions.

Description

I observed a geometric inconsistency issue in RTX lidar point clouds when the sensor undergoes rapid motion.
The issue can be observed directly in the Isaac Sim viewport using the DebugDrawPointCloud visualization. During aggressive vehicle motion, point clouds no longer remain spatially aligned with static landmark surfaces and instead appear to slide or smear relative to the geometry.

The issue was also reproduced through ROS2 point cloud publication. I recorded a ROS2 bag containing:

• point cloud messages
• lidar odometry (sensor pose with respect to the world frame)
  I then replayed the bag and visualized the data in RViz using a custom ROS2 node that transforms each point cloud into the world frame using the corresponding sensor pose at each frame. The same geometric inconsistency was observed in RViz as well.

Step to reproduce

I created a minimal reproducible repository containing the simulation setup and instructions: https://github.com/yuheisugano/isaacsim-lidar-vehicle-repro.git
Please clone the repository and follow the instructions in README.md.

Isaac Sim Version

• 5.1.0
• 6.0-dev2

Operating System (OS)

• Ubuntu 24.04.3 LTS

GPU Name

• NVIDIA GeForce RTX 4090

GPU Driver and CUDA versions

• Driver Version: 580.159.03
• CUDA Version: 13.0

Logs

A log file is in logs/ within the reproduction repository.

Additional Information

• The goal is to simulate lidar point clouds for moving-vehicle dataset generation in SLAM / odometry experiments.
• The vehicle motion is simulated using VehicleSDK in order to model more realistic vehicle dynamics, including tire-ground interaction and slipping behavior.
• I experimented with several potentially relevant parameters, including:
  • simulation_app.enable_motion_bvh
  • sim_time.physics_dt
  • sim_time.rendering_dt
  • omni:sensor:Core:outputMotionCompensationState
  • omni:sensor:Core:scanRateBaseHz
  • omni:sensor:Core:patternFiringRateHz
    However, I was unable to resolve the issue.
• The issue appears more prominently when:
  • vehicle speed increases
  • steering amplitude becomes larger
  • sensor motion becomes more aggressive
• Screen recordings of the simulation and rviz visualization are in media/ within the reproduction repository.

Please let me know if additional logs, videos, or reduced test cases would be helpful.

[michaellin6]

Hi @yuheisugano, Thank you for positing this issue and also for sharing the repro.

On first thought if your rendering rate matches your lidar scan rate exactly — both at 10 Hz. With rendering_dt: 0.1 and scanRateBaseHz: 10.0 (the Example_Rotary default), one full 360° sweep is built from a single GPU render frame. All rays are cast from the sensor's pose at that one instant, but the motion compensation pipeline treats them as if they were emitted progressively across the 100 ms sweep at interpolated poses. At ~5 m/s with steering, that 100 ms of "phantom" pose variation introduces the sliding/smearing you're seeing.
If you have not tried it, try rendering_dt = 0.01 (100 Hz) or rendering_dt = 0.02 (50 Hz) while keeping scanRateBaseHz: 10.0. This gives the sensor 5–10 distinct poses per sweep, so raycasts actually come from different positions. In the meantime, I am also trying your repro and checking for what may be going on.

[michaellin6]

Following up on this — I built a minimal kinematic version of your scene (no Vehicle SDK, no ROS2, just an Xform translating the lidar past a static wall at 8 m/s with sinusoidal yaw) and I'm getting the same sliding behavior. The cause is your sim_time.rendering_dt: 0.1 exactly matching the Example_Rotary scan rate of 10 Hz — one render per full scan rotation, which is the worst case for the per-frame consumer pattern. The annotator IsaacExtractRTXSensorPointCloudNoAccumulator returns points referenced to the sensor pose at scan time, but the lidar prim's USD ComputeLocalToWorldTransform at the moment your publisher reads it is already one rendering_dt later — so the cloud ends up offset by ~2 × rendering_dt × sensor_velocity in the direction of motion (about 1.6 m for your parameters).

Fix: drop rendering_dt to 0.005 (200 Hz, 20 renders per scan rotation). Everything else — enable_motion_bvh: true, outputMotionCompensationState: COMPENSATED, outputFrameOfReference: SENSOR, zero_delay.kit, the ROS2 stamp source — can stay as you have it.

Numbers from a kinematic isolation test (8 m/s longitudinal + ±0.4 rad sinusoidal yaw, wall at world x = 17.9 m, consumer reads IsaacExtractRTXSensorPointCloudNoAccumulator.data and applies Xformable.ComputeLocalToWorldTransform(0.0)):

rendering_dt = 0.100 s  -> wall centroid mean = 19.64 m   (residual +1.74 m, max +1.89 m)
rendering_dt = 0.005 s  -> wall centroid mean = 18.10 m   (residual +0.19 m, max +0.32 m)

The remaining ~7 cm in fix-mode is the wall's own 0.2 m thickness averaged front/back as the sensor passes through.

Background reading: the RTX Sensor Annotators page describes the same artifact in the "dragging" warning. If you'd rather not raise your rendering rate, the alternative is to switch from IsaacExtractRTXSensorPointCloudNoAccumulator to IsaacCreateRTXLidarScanBuffer and consume its Transform output (lidar→world at end-of-scan) instead of Xformable.ComputeLocalToWorldTransform — that output is the correct reference for the buffered data. The per-frame extractor doesn't expose a Transform output, so the consumer has to obtain the lidar pose another way, and reading it from USD at the wrong time is what produces the sliding.

Attached are both my simplified repro and fix scripts.
fix.py
repro.py

[yuheisugano]

Hi @michaellin6, Thank you so much for the detailed investigation and repro scripts. The explanation regarding the synchronization between rendering rate and scan generation was very helpful.

After testing the suggested fixes, I now understand that reducing rendering_dt does significantly improve the geometric consistency of the scan.

However, the remaining issue for my use case is the worst-case distortion (e.g., the max residual +0.32m wall position in the shared case). Even after accounting for wall thickness, the remaining error can still become noticeable during aggressive sensor motion, especially under high angular velocity.

I also tested:

• IsaacCreateRTXLidarScanBuffer
• consuming the annotator Transform output directly
• much smaller values such as rendering_dt = physics_dt = 0.00001 s

The overall behavior improves substantially with smaller rendering_dt, but some shadowing/blurring artifacts still remain in aggressive-motion scenarios.

Since my primary use case is SLAM/odometry research, I would like to prioritize geometric consistency over simulation speed.
Therefore, I would like to ask:
Is there any way to generate a geometrically consistent or artifact-free scan?
For example:

• generating one vertical scan column per rendering frame and manually accumulating them,
• generating an instantaneous whole scan within a single rendering frame (without motion distortion),
• or another method that guarantees geometrically consistent scans even under aggressive motion.

I understand that an instantaneous scan may not be physically realistic for a rotating LiDAR, but such an option would be extremely valuable for controlled perception experiments and pose-estimation benchmarking.

Thank you again for your help and detailed investigation. I really appreciate the time and effort you put into reproducing and analyzing the issue.

[yuheisugano]

Hi @michaellin6, sorry to follow up again.

I wanted to kindly check whether you might have any further thoughts on my follow-up question above. I understand you may be busy, but this issue is still blocking my RTX lidar simulation experiments, so any guidance or suggestions would be greatly appreciated.

Thank you again for your help.

[michaellin6]

Hi @yuheisugano, sorry for the slow reply, I am checking with the internal team that works on the LiDAR motion compensation component. @adevalla-nvidia Can you help follow up on this?

[michaellin6]

internal ticket: 6313423

[michaellin6]

Update — we root-caused this and there's a clean fix.

The smear isn't a rendering or motion-compensation bug; it comes from how the cloud is transformed on the consumer side. When you read the cloud in the sensor frame and transform it to world using the lidar prim's read-time USD pose, that single pose doesn't match the per-ray emission poses the sweep was motion-compensated against. During fast motion the mismatch shows up as the ~2·v·dt slide you measured.

Fix: have the RTX pipeline emit the cloud already in the WORLD frame, and consume it verbatim (no USD-pose transform). In WORLD mode the engine resolves every return with its own motion-compensated per-ray pose, so the timing offset never enters:

attributes = {
    "omni:sensor:Core:outputMotionCompensationState": "COMPENSATED",
    "omni:sensor:Core:outputFrameOfReference": "WORLD",   # <-- the key change
    "omni:sensor:Core:scanType": "ROTARY",
    "omni:sensor:Core:elementsCoordsType": "CARTESIAN",
}
# consumer: points are already world-frame; do NOT multiply by the prim pose.

On a minimal repro (kinematic lidar carrier, 8 m/s + sine steering, past a static wall) this takes the wall-return residual from +2.36 m (naive read-time-pose transform) down to ~0 m (mean +0.0003 m, max 0.016 m) while the sensor is approaching the wall — i.e. the scans are geometrically consistent, which is what matters for SLAM/odometry.

I've attached a minimal runnable version fix_worldframe.py that sets WORLD-frame and has no point cloud smear (→ ~0 m).

Two notes:

• If you use the Example_Rotary config and want to measure pure geometry, zero its baked-in noise terms (rangeAccuracyM, rangeResolutionM, azimuthErrorMean/Std, elevationErrorMean/Std) — they add a small residual on top.
• If for some reason you must keep a sensor-frame consumer, lowering rendering_dt well below the scan period (e.g. 0.005 s at a 10 Hz scan) reduces the slide ~9× but does not eliminate it — a smaller dt just shrinks the same read-time-pose error (it narrows the accumulation window), it doesn't remove it. The WORLD-frame path is the real fix.

Let us know if this resolves it on your vehicle setup.

[michaellin6]

fix_worldframe.py