LeRobot

ROS 2 Jazzy robotics learning project.

Completed Milestones

Milestone 1: SO101 Robot Visualization

• ROS 2 Jazzy installation
• Workspace creation
• SO101 robot description package
• Robot visualization in RViz

Milestone 2: Camera + YOLO Object Detection

• USB camera integration
• ROS image topic subscription
• OpenCV image processing
• YOLOv8 real-time object detection

Objects currently detected:

• Person
• Cell phone
• Bottle
• Chair
• Laptop
• Other YOLOv8 supported classes

Milestone 3: SO101 Gazebo Pick-and-Place with Wrist Camera

• SO101 arm spawned in Gazebo
• ROS 2 controllers active
• Gripper controller working
• 2 cm cube spawned and stable
• Virtual cube attach/release system working
• Automatic pick-place sequence working
• One-command full pick-place demo working
• Wrist camera mounted near gripper
• Gazebo wrist camera topic publishing
• ROS 2 camera bridge working
• Wrist camera image visible in rqt_image_view
• Full pick-place demo works while wrist camera streams

Workspace Structure

ros2_ws/
├── src/
│   ├── camera_vision/
│   ├── so101_description/
│   ├── so101_gazebo/
│   └── so101_moveit_config/

Requirements

• Ubuntu 24.04
• ROS 2 Jazzy
• Gazebo Sim
• Python 3.12
• OpenCV 4.8.1
• Ultralytics YOLOv8

Running Camera Detection

Terminal 1:

ros2 run usb_cam usb_cam_node_exe

Terminal 2:

source ~/ai_projects/yolo_env/bin/activate
python ~/ros2_ws/src/camera_vision/camera_vision/camera_viewer.py

Running SO101 Gazebo Pick-and-Place with Wrist Camera

Terminal 1:

source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash

ros2 launch so101_gazebo full_auto_pick_place_with_camera.launch.py

Terminal 2, to view wrist camera:

source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash

ros2 run rqt_image_view rqt_image_view /wrist_camera/image

Important ROS Topics

/wrist_camera/image
/wrist_camera/camera_info
/pick_place_state
/joint_states
/arm_controller/follow_joint_trajectory
/gripper_controller/follow_joint_trajectory

Current Working Safe Poses

pick_pose  = [0.0, -0.5, 0.5, -0.3, 0.0]
place_pose = [0.6, -0.5, 0.5, -0.3, 0.0]
home       = [0.0, 0.0, 0.0, 0.0, 0.0]

Current Status

Working:

• Real-time YOLO camera detection
• SO101 Gazebo automatic pick-place
• Wrist camera image streaming from Gazebo to ROS 2

Next goal:

• Detect cube from wrist camera image
• Publish detected object center as a ROS 2 topic
• Later convert image pixel + depth into 3D robot coordinates

---

Final Gazebo V1 Demo: Camera-Centered Pick and Place

This launch runs the complete Gazebo simulation pipeline:

• Spawn SO101 robot
• Start controllers
• Start wrist camera bridge
• Move cube into wrist camera view
• Detect red cube using OpenCV color detection
• Compute visual servo error
• Center cube in camera view using shoulder pan and wrist flex
• Trigger automatic pick-place after centering
• Attach cube virtually
• Move cube to place pose
• Release cube
• Finish sequence

Run:

source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash

ros2 launch so101_gazebo vision_center_then_pick.launch.py

source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash

ros2 launch so101_gazebo final_so101_gazebo_pick_place.launch.py

V3: MoveIt + Wrist Camera + Gazebo Pick-Place

Final command

cd ~/ros2_ws
source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash

ros2 launch so101_moveit_cpp v3_final_moveit_pick_place.launch.py

---

V4: Camera-Derived Cube Pose Pick-Place

Final command

cd ~/ros2_ws
source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash

ros2 launch so101_moveit_cpp v4_camera_derived_pick_place.launch.py

Gazebo SO101 robot ↓ wrist camera image ↓ red cube detector ↓ /camera_cube_error ↓ V4 camera-derived cube pose publisher ↓ /pick_cube_pose ↓ MoveIt shoulder pan refinement ↓ grasp, attach, lift, place, release

V4 CAMERA-DERIVED /pick_cube_pose CAMERA ALIGNMENT PASSED V4 camera-derived pose after alignment V4 refining shoulder_pan PAN_REFINE_CAMERA_DERIVED PRE_GRASP_CAMERA_DERIVED CUBE ATTACHED CUBE LIFTED CUBE RELEASED TO GROUND V4 MOVEIT CAMERA-DERIVED PICK COMPLETE

---

V5: Random Cube Position Pick Demo

Final command

cd ~/ros2_ws
source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash

ros2 launch so101_moveit_cpp v5_random_cube_position_pick.launch.py cube_position:=RANDOM

Manual test commands

Run LEFT cube position:

ros2 launch so101_moveit_cpp v5_random_cube_position_pick.launch.py cube_position:=LEFT

Run CENTER cube position:

ros2 launch so101_moveit_cpp v5_random_cube_position_pick.launch.py cube_position:=CENTER

Run RIGHT cube position:

ros2 launch so101_moveit_cpp v5_random_cube_position_pick.launch.py cube_position:=RIGHT

Run RANDOM cube position:

ros2 launch so101_moveit_cpp v5_random_cube_position_pick.launch.py cube_position:=RANDOM

What this demo adds over V4

V4 used camera-derived cube pose correction, but the cube still started from the same position most of the time.

V5 adds visible cube position variation.

The cube can now start from LEFT, CENTER, RIGHT, or RANDOM positions. The robot uses camera detection and MoveIt shoulder-pan refinement to adapt to the cube position.

V5 pipeline

Gazebo cube position selector
        ↓
LEFT / CENTER / RIGHT / RANDOM cube pose
        ↓
wrist camera detects red cube
        ↓
/camera_cube_error
        ↓
camera-derived /pick_cube_pose
        ↓
MoveIt shoulder_pan refinement
        ↓
grasp, attach, lift, place, release

Confirmed working behavior

V5 has been tested successfully with:

V5 CUBE POSITION SELECTED: LEFT
CAMERA ALIGNMENT PASSED
V4 refining shoulder_pan from -0.303 to -0.231
CUBE ATTACHED
CUBE LIFTED
CUBE RELEASED TO GROUND
V4 MOVEIT CAMERA-DERIVED PICK COMPLETE

And:

V5 CUBE POSITION SELECTED: RIGHT
CAMERA ALIGNMENT PASSED
V4 refining shoulder_pan from -0.303 to -0.412
CUBE ATTACHED
CUBE LIFTED
CUBE RELEASED TO GROUND
V4 MOVEIT CAMERA-DERIVED PICK COMPLETE

Current V5 status

• LEFT cube position: working
• CENTER cube position: working
• RIGHT cube position: working
• RANDOM cube position: working
• Wrist camera detection: working
• Camera-derived pose correction: working
• MoveIt pan refinement: working
• Cube attach, lift, release: working

---

Final Demo: SO101 Red Target Pick and Drop

This is the final Gazebo and MoveIt demo checkpoint.

The robot uses a wrist camera to detect the red target cube, ignores blue and green distractor cubes, picks the red cube, lifts it, and drops it on the black drop-zone pad.

Final launch command

cd ~/ros2_ws
source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash

ros2 launch so101_moveit_cpp v10_final_demo.launch.py cube_position:=RANDOM

Manual test commands

Run the red cube at LEFT:

ros2 launch so101_moveit_cpp v10_final_demo.launch.py cube_position:=LEFT

Run the red cube at CENTER:

ros2 launch so101_moveit_cpp v10_final_demo.launch.py cube_position:=CENTER

Run the red cube at RIGHT:

ros2 launch so101_moveit_cpp v10_final_demo.launch.py cube_position:=RIGHT

Run random red cube position:

ros2 launch so101_moveit_cpp v10_final_demo.launch.py cube_position:=RANDOM

What the final demo shows

Gazebo world starts
Robot spawns
Red target cube starts at LEFT, CENTER, RIGHT, or RANDOM
Blue and green cubes stay as distractors
Wrist camera searches for the red target
Detector publishes red target center error
Camera-derived target pose is published
MoveIt refines shoulder pan
Robot moves to pre-grasp
Cube attaches to gripper
Robot lifts the cube
Robot moves to drop area
Cube releases onto the black drop-zone pad

Expected success logs

V9 RED TARGET POSITION SELECTED
V9 RED TARGET MOVE SERVICE RESULT: success=True
CAMERA ALIGNMENT PASSED
V9 red target pose after alignment
V9 refining shoulder_pan
CUBE ATTACHED
CUBE LIFTED
CUBE RELEASED
V9 RED TARGET PICK AND DROP COMPLETE

Known warnings that can be ignored

These warnings may appear during the demo and are not blockers if the final success log appears:

No 3D sensor plugin(s) defined for octomap updates
No state update received within 100ms of system clock

Final working status

• Gazebo robot visible: working
• MoveIt robot motion: working
• Wrist camera stream: working
• Red target detection: working
• Blue and green distractor cubes: working
• Random red cube position: working
• Camera-derived pose correction: working
• Gripper attach and release: working
• Black drop-zone pad: working
• Final pick and drop sequence: working