Non Repetitive LiDAR support

[justinwood97]

Hi, I'm currently running Isaac Sim 5.1.0 on an RTX3090.
My organisation is currently using the Livox Mid 360, and there was a need for me to replicate it within Isaac Sim for accurate Sim2Real demonstration. However, I quickly discovered how little support there is for non-repetitive LiDAR.

Using the PhysX LiDAR template and a real life recording of the Livox Mid 360's azimuth and zenith, I was able to replicate the Livox Mid 360 in Isaac Sim completely, including the non-repetitive pattern. But because this uses the PhysX api, I worry about whether or not this will continue to be supported in the future + the other general benefits of using RTX based lidar.

I tried my hand at replicating the Livox Mid 360 using the pre-existing RTX Lidar API, but so far, I haven't gotten very far. Is there no way to bootstrap the existing RTX Lidar Api in a similar fashion to the PhysX API to use the non-repetitive scanning pattern? Does Isaac Sim 6.0 currently support it? Are there future plans to support non-repetitive scanning LiDAR?

[vick-yu]

We are looking into this and will update you here.

[vick-yu]

Great question, and impressive work getting the Livox Mid 360 working via the PhysX LiDAR path.

The RTX LiDAR API does support non-repetitive firing patterns -- it just requires a bit more manual setup compared to PhysX. The key insight for Livox sensors is that their firing pattern is not frame-recurring (i.e., it doesn't reset every frame) but rather repeats after N frames. The RTX API handles this natively via emitterStateArrays.

How to Set It Up

The approach is to extract the real per-frame firing patterns from your recorded data and encode them as a sequence of emitterStateArrays:

1. Extract per-frame patterns from your real Livox recording -- save the azimuth, elevation, fire time, and channel ID for each beam in each frame.

2. Find the repeat period -- check after how many frames you see the same pattern repeat (there will be some noise, so look for approximate periodicity). This period becomes your emitterStateCount.

3. Build one emitterStateArray per frame in the cycle. Each entry needs at minimum:

   • azimuthDeg
   • elevationDeg
   • fireTimeNs
   • channelId
     (other fields are optional)

4. Set stateResolutionStep to 1 -- this tells the simulator to advance to the next emitter state array on every frame, working through your sequence in order.

5. Average across cycles to reduce noise -- for example, if the pattern repeats every 20 frames, average together frames 1, 21, 41, 61, ... to get a cleaner emitter state for "slot 0", and likewise for slots 1-19.

Concrete Example

If your Livox Mid 360 data shows a 20-frame repeat cycle:

• emitterStateCount = 20
• You define 20 emitterStateArrays (one per frame in the cycle)
• stateResolutionStep = 1

The simulator will use array[0] on frame 0, array[1] on frame 1, ..., array[19] on frame 19, then loop back to array[0] -- matching the real sensor's behavior.

If Livox doesn't publish their official firing pattern tables, the above approach of extracting it from real data is the recommended path.

Isaac Sim 6.0

The RTX LiDAR API with emitterStateArrays is supported in Isaac Sim 6.0 as well, so your config should be portable. The PhysX path will continue to work for now, but the RTX path is the better long-term bet for sim fidelity and future support.

Let us know if you run into issues constructing the emitter state arrays -- happy to help debug the config.

[justinwood97]

Thanks for the great answer, this should be plenty for me to get started on.
I'll keep you posted on whatever issues I stumble upon.

[justinwood97]

So I implemented the LiDAR last week but only got around to writing this today.

TLDR: I got the RTX LiDAR working, but face issues with stability and output data. I also got pulled off to another project, so development is suspended temporarily.

Following vick-yu's advice worked pretty well.

1. My Livox scan data had around 800k rows of data and after checking, I only found meaningful repetition at around 500k rows (which covered around 99% the scan area of the original 800k rows). But even 500k points = 500k emitters for an RTX LiDAR, which had several issues:

• EmitterState count : to fit 500k emitters into one LiDAR, I first tried just spreading it all out across 100 emitterStates (5000 points per emitterState). But, Isaac Sim starts lagging real bad after even 50 emitterStates in one LiDAR prim. I ended up running with 25 emitterStates, which didn't cause any noticeable lag.
• Hydra API Limits : 500k emitters is a lot of data. The API that attaches emitterStates to the LiDAR prim has a limit of 5mb per LiDAR. According to my calculations, this was around 200k emitters per LiDAR prim, also dependent on whether you add in the optional parameters for each emitterState.

2. Luckily for me, the Livox scan data clearly had 4 independent laser beams shooting out at different zeniths and azimuths but at the same timeframe as each other. So I split the LiDAR up into 4 separate LiDAR prims, with each prim representing one laser beam, taking 125k emitters each, which fit under the 5mb Hydra limit. Positioning them at the same world transform and running the sim produced a pretty realistic looking Livox mid360 scan pattern.

3. I attached ROS2 Lidar helpers to each prim and streamed the data to RVIZ2 for visualisation. However, this caused the aforementioned instability. Isaac Sim would often crash when running the sim after attaching the helpers with the following error repeated over and over again.

CUDA error 1 in ../../../internal/include/omni/sensors/cuda/CudaHelperMem.h:324:invalid argument (cudaMemcpyAsync((void*)dstPtr, (void*)srcPtr, sizeInBytes, kind, cudaStream))

Occasionally, I'll get lucky and the sim runs and the points do get processed and published properly. This doesn't last though - if I stopped the sim and ran it again, it crashes with the error above as well.

4. While the LiDAR prims were up and running, I took the opportunity to inspect the points returned. Since the real world Livox sensor returns 200k points per second and I had 500k points total, it meant that I had 2.5 seconds worth of emitterStates to go through.
   Each emitterState contained 5000 emitters and with 4 LiDAR prims running at 10Hz, I expected this to be 5000 emitters x 4 prims x 10 Hz = 200k points per second.

I published each LiDAR prim to their own topic and it turns out each prim was returning around 50k points per frame. Since the sim ran at 60Hz, this meant that in one second, the actual point return rate was 60k x 4 x 60 = 14,400,000 points per second. I also did notice that it didn't always return the exact same amount of points per frame - it would also fluctuate greatly. At the start, they would return 50k points per frame and towards the end, they start returning either no points or around 5000 points per frame before looping back to returning 50k points per frame and starting this "cycle" over again.

Here are some images of my LiDAR point cloud patterns. Note that each LiDAR prim is assigned a different colour.

When all LiDAR prims are returning 50k points on the same frame, earlier in the "cycle"

When only two LiDAR prims are returning 50k points on the same frame and for some reason the other 2 aren't returning anything, mid "cycle"

When the LiDAR prims start returning only 5000 points per frame (which is expected, just the timing isn't right), late "cycle"

Questions

1. Instability

• What causes the instability with Isaac Sim?
• Could it be that I misconfigured my LiDAR prims somewhere and it's causing a CUDA error inconsistently?

2. Point cloud return rate

• How can I limit the LiDAR prims to strictly return 200k points per second?
• My current theory is that we're only meant to use emitterStates for 1 second of simulation, but since my emitterStates span 2.5 seconds, Isaac Sim cramps my emitterStates into 1 second of runtime, which results in the wonky return rate. I haven't been able to test this, but if this is true then the only true solution would be to cut down to 200k emitters total, which wouldn't be optimal.
• I tried playing around with different scanRateBaseHz, reportRateBaseHz and tick rate, but didn't manage to solve anything.

If you could point me in the right direction, I'd be happy to disclose as much of my setup as I can.
Here's the LiDAR config parameter I used for each LiDAR prim, with emitterStates omitted for brevity. The LiDAR prim names are different (A,B,C,D) but they all share the exact same configurations.

Config for: /World/Livox_Mid360_A
==================================================
  omni:sensor:Core:aspectRatio = 1.0
  omni:sensor:Core:auxOutputType = NONE
  omni:sensor:Core:avgPowerW = 0.0020000000949949026
  omni:sensor:Core:azimuthErrorMean = 0.0
  omni:sensor:Core:azimuthErrorStd = 0.05000000074505806
  omni:sensor:Core:beamWaistHorM = 0.0020000000949949026
  omni:sensor:Core:beamWaistVerM = 0.0020000000949949026
  omni:sensor:Core:bitDepthResolution = 65536.0
  omni:sensor:Core:calibrationGain = 40.0
  omni:sensor:Core:customFrameOfReferenceTrafo = [0, 0, 0, 0, 0, 0]
  omni:sensor:Core:divergenceHorDeg = 0.0
  omni:sensor:Core:divergenceVerDeg = 0.0
  omni:sensor:Core:effectiveApertureSizeM = 0.009999999776482582
  omni:sensor:Core:elementsCoordsType = SPHERICAL
  omni:sensor:Core:elevationErrorMean = 0.0
  omni:sensor:Core:elevationErrorStd = 0.05000000074505806
  omni:sensor:Core:farRangeM = 40.0
  omni:sensor:Core:focusDistM = 0.15000000596046448
  omni:sensor:Core:intensityMappingDecoding = []
  omni:sensor:Core:intensityMappingEncoding = []
  omni:sensor:Core:intensityMappingType = LINEAR
  omni:sensor:Core:intensityProcessing = NORMALIZATION
  omni:sensor:Core:intensityScalePercent = 255.0
  omni:sensor:Core:maxAzimuthROI = 400.0
  omni:sensor:Core:maxReturns = 1
  omni:sensor:Core:minAzimuthROI = 400.0
  omni:sensor:Core:minDistBetweenEchosM = 0.4000000059604645
  omni:sensor:Core:minReflectance = 0.10000000149011612
  omni:sensor:Core:minReflectionRangeM = 10.0
  omni:sensor:Core:Msquared = 1.399999976158142
  omni:sensor:Core:nearRangeM = 0.10000000149011612
  omni:sensor:Core:numberOfChannels = 5000
  omni:sensor:Core:numberOfEmitters = 5000
  omni:sensor:Core:numLines = 1
  omni:sensor:Core:numRaysPerLine = [5000]
  omni:sensor:Core:originErrorMean = [0, 0, 0]
  omni:sensor:Core:originErrorStd = [0, 0, 0]
  omni:sensor:Core:outputFrameOfReference = SENSOR
  omni:sensor:Core:outputMotionCompensationState = NONCOMPENSATED
  omni:sensor:Core:pixelPitch = 40.0
  omni:sensor:Core:pulseTimeNs = 6
  omni:sensor:Core:quantumEfficiency = 0.8999999761581421
  omni:sensor:Core:rangeAccuracyM = 0.019999999552965164
  omni:sensor:Core:rangeCount = 1
  omni:sensor:Core:rangeOffsetM = 0.0
  omni:sensor:Core:rangeResolutionM = 0.0020000000949949026
  omni:sensor:Core:rangesMaxM = []
  omni:sensor:Core:rangesMinM = []
  omni:sensor:Core:rayType = IDEALIZED
  omni:sensor:Core:reflectionPowerFraction = 0.5
  omni:sensor:Core:reportRateBaseHz = 10
  omni:sensor:Core:rotationDirection = CW
  omni:sensor:Core:scanRateBaseHz = 10
  omni:sensor:Core:scanType = SOLID_STATE
  omni:sensor:Core:skipDroppingInvalidPoints = True
  omni:sensor:Core:startAzimuthOffsetDeg = 0.0
  omni:sensor:Core:stateResolutionStep = 1
  omni:sensor:Core:transmissionPowerFraction = 1.0
  omni:sensor:Core:validEndAzimuthDeg = 360.0
  omni:sensor:Core:validStartAzimuthDeg = 0.0
  omni:sensor:Core:waveLengthNm = 903.0
  omni:sensor:encryption:params = []
  omni:sensor:marketName = Generic
  omni:sensor:modelName = Livox_Mid360_A
  omni:sensor:modelVendor = Livox
  omni:sensor:modelVersion = 1.0.0
  omni:sensor:tickRate = 10.0
  purpose = default
  visibility = inherited

Each LiDAR prim has 25 emitterStates that contain 5000 emitters each. A 360° sweep is roughly 10 emitterStates, but due to Mid360's non repetitive nature, it takes 25 emitterStates for the pattern to show no further meaningful coverage and to loop nicely.

[vick-yu]

Great progress -- splitting the Livox Mid 360 into 4 prims to stay under the Hydra 5 MB limit and using 25 emitterStates is a clever approach, and the point cloud screenshots confirm the firing pattern itself is correct.

There are two separate issues here. Let me break them down:

---

1. Point Return Rate (14.4M pts/sec instead of 200K)

The root cause is a render rate vs. scan rate mismatch specific to Isaac Sim 5.1.0.

On 5.1.0, multi-tick rendering is not enabled by default. This means omni:sensor:tickRate is effectively ignored -- every RTX sensor renders on every simulation frame, regardless of the tick rate you set. With your sim running at 60 Hz but scanRateBaseHz = 10, the sensor fires 6x more often than intended, which is why you see ~50K points per prim per frame instead of 5K.

The fluctuating point counts (50K -> 5K -> 0 -> 50K cycling) are a direct consequence of this mismatch: the sensor advances through your 25 emitterStates faster than intended, causing frame-boundary artifacts when it wraps from state 24 back to state 0.

Fix options (5.1.0):

• Set publishStepSize = 6 on each ROS2 RTX LiDAR Helper node. This won't fix the internal firing rate, but it will throttle ROS2 publishing to roughly 10 Hz (60/6). The underlying sensor still renders every frame, so the point data may still contain accumulation artifacts.

• Match the simulation frame rate to the scan rate by calling RenderingManager.set_dt(1.0/10.0) -- but this limits your entire simulation to 10 Hz, which is usually impractical.

Better fix: upgrade to Isaac Sim 6.0+. Multi-tick rendering is on by default in 6.0, and tickRate = 10 will actually limit the sensor to 10 Hz rendering regardless of the sim frame rate. This is the correct architectural solution for your setup. See the Multi-Tick Rendering docs for details.

Key from the docs: "Before Isaac Sim 6.0, every camera and RTX sensor rendered at the simulation frame rate."

Also note: reportRateBaseHz was removed in the 6.0 sensor schema, so if you upgrade, remove that attribute from your configuration.

---

2. CUDA Crash (cudaMemcpyAsync error 1)

The CUDA error 1 (invalid argument) in CudaHelperMem.h:324 during cudaMemcpyAsync is a known issue when multiple RTX LiDAR prims with large emitter counts share GPU resources. With 4 prims each returning ~50K points per frame at 60 Hz, the GPU return buffer management hits race conditions.

This is related to a known crash pattern with RTX LiDAR + ROS2.

Mitigations:

1. Try launching with --/app/sensors/nv/lidar/outputBufferOnGPU=false -- this forces CPU-side buffering for LiDAR returns, which avoids the GPU memory races at the cost of some throughput.

2. Fixing the point rate issue first will likely help stability -- once the sensor actually ticks at 10 Hz instead of 60 Hz, the GPU load drops by 6x and the buffer pressure that triggers the crash largely goes away.

3. If the crash persists even with correct tick rates, try initializing ROS2 helpers one at a time with a few warm-up frames between each (the race condition is most likely during buffer allocation at startup).

---

Summary

Issue	Root Cause	Best Fix
14.4M pts/sec	5.1.0 ignores tickRate, renders at 60 Hz	Upgrade to 6.0+ (multi-tick rendering)
Point count fluctuation	EmitterState cycling at 6x intended speed	Same -- correct tick rate fixes this
CUDA crash	GPU buffer races with 4 prims at 60 Hz	outputBufferOnGPU=false + correct tick rate

Your emitterState configuration itself looks correct -- 25 states with 5000 emitters each, stateResolutionStep = 1, scanType = SOLID_STATE. The issues are all downstream of the timing mismatch.

When you get back to this project, I'd recommend starting with the 6.0 upgrade. The reportRateBaseHz attribute was removed, but your scan pattern setup (emitterStates + stateResolutionStep) carries over directly. Let us know how it goes!

[vick-yu]

@justinwood97, do you have any progress?

Here are some additional details based on further investigation:

On the 5 MiB emitterState budget. The per-prim limit is enforced in the RTX Hydra
render bridge when it serializes all sensor attributes into the flat buffer the renderer
reads. The budget covers every attribute on the prim, and the emitterState arrays dominate:
roughly emitterStateCount * numberOfEmitters * 16 bytes for the four required fields
(azimuthDeg, elevationDeg, fireTimeNs, channelId). With your config of 5000 emitters per
state and required fields only, the crossover is at approximately 65 emitter states (~325k
total emitters), not the ~40 states implied by a flat "200k total emitters" rule. Optional
fields (bank, rangeId, offsets) reduce this further proportionally. One thing worth knowing:
when the budget is exceeded, the error is logged to an internal RTX channel (not the main
console), and the sensor is silently torn down and recreated every render frame -- which is
why 100 emitter states looked like severe lag rather than a clean error message. Your
25-state working config serializes to about 1.9 MiB per prim, comfortably within the limit.

On the point-rate cycling. Your "2.5 seconds of emitterStates crammed into 1 second"
theory was close, but the state counter itself still advances at the configured scan rate --
it is not compressed. What happens is that the renderer fires at 60 Hz regardless of
tickRate, and renders whichever emitter state is currently active, six times per state
period. The 50k -> 5k -> 0 -> 50k cycling you observed is the transition artifact at the
boundary between emitter states at 60 Hz render rate. Once you are on 6.0+, multi-tick
rendering limits each prim to 10 Hz, and each render tick will return exactly the expected
point count for the active state.

Additional 5.1.0 ROS2 workaround. The ROS2 RTX Lidar Helper has an inputs:fullScan
flag (True/False) that accumulates partial frames until a complete scan boundary is
reached before publishing. This is cleaner than frameSkipCount alone since it publishes
on scan completion rather than on a fixed frame interval. Note that fullScan only works
with type = point_cloud; if you are publishing laser_scan messages, frameSkipCount
is the right approach.

og.Controller.set(og.Controller.attribute("/ROS_LidarHelper.inputs:fullScan"), True)
og.Controller.set(og.Controller.attribute("/ROS_LidarHelper.inputs:frameSkipCount"), 5)

[vick-yu]

We haven't heard back since the last update, so we'll go ahead and close this for now. If you're still hitting issues when you return to this project, feel free to reopen this issue directly using the 'Reopen issue' button, or open a new one referencing this thread so we have full context. Thanks again!