Project 01 / Human-to-Robot Teleoperation / 2026

Unified Human-to-Robot Teleoperation Platform

A perception-to-actuation system that mirrors human finger and arm motion as dexterous-hand commands and Cartesian TCP targets across half-humanoid and single-arm robots.

21 → 6Hand landmarks to channels

1 kHzIK + impedance control

01Human Motion CaptureWebcam landmarks + Vive poses
02Motion RetargetingFinger channels + robot TCPs
03Gate & StreamFreshest valid targets only
04Robot ActuationCAN hand + impedance arm

01 / Hand Pose Pipeline

Camera landmarks become protected finger motion

On the FR3 / RealHand L6 setup, the hand path retargets a camera-observed human hand into six actuator commands without requiring user-specific hand-size calibration.

Perception and retargeting

Keep only the newest camera frame and detect 21 MediaPipe landmarks.
Map relative finger-bone angles to five flexion channels plus thumb rotation.
Normalize commands to 0–1 and apply EMA smoothing before streaming.
Enter execute mode only while a hand remains visible.

Safe physical actuation

Interpolate timestamped targets in a 120 Hz hand-control loop.
Return to idle after 0.35 s without a detected hand.
Back off joints under excessive force and enforce configured angle limits.
Reorder and quantize six channels for the RealHand L6 over 1 Mbit/s SocketCAN.

02 / TCP Teleoperation Pipeline

Vive motion becomes fresh Cartesian robot targets

The arm path uses one Cartesian contract for synchronized half-humanoid hands or a single 6-DoF arm, keeping robot-specific inverse kinematics and dynamics out of the operator client. It is supported by my Real-Time Compliant Half-Humanoid Control project, which provides the 1 kHz whole-body IK and Cartesian impedance foundation beneath teleoperation.

Freshness-first pose path

Reference, reorient, smooth, and continuity-correct Vive tracker poses.
Use conflated ZeroMQ transport and drain every queue to its newest frame.
Apply configuration-driven tracker-to-TCP and tracking-to-base transforms.
Gate execution off by default while keeping the stream ready.

Driver-side motion and feedback

Stream typed targets and receive state plus IK diagnostics over bidirectional gRPC.
Reject stale targets and large jumps; use minimum-jerk engagement for new targets.
Run driver-side IK at 1 kHz and hold any inactive arm at its current pose.
Feed a 1 kHz gravity-, friction-, and impedance-compensated torque loop.

03 / Shared Design

Safety and observability connect both pipelines

Both paths separate motion capture from execution, favor fresh data over stale backlog, and expose the state that the hardware actually accepted. Hand visibility gates finger motion, force protection limits contact, arm targets pass freshness and jump checks, and driver faults return the robot to a controlled hold or compliant state. End-to-end logs preserve perception, requested targets, accepted targets, measured state, and timing so tracking noise, solver error, and physical following error can be diagnosed independently.