Real-Time Chunking (RTC)

Real-Time Chunking (RTC) allows large, flow-matching based robotic policies, such as Pi0, Pi0.5, and SmolVLA, to produce smooth, continuous, and reactive motion despite having high inference latency. LeRobot provides the original inference-time guided mode and, for compatible Pi05 checkpoints, training-time action conditioning with cheap hard-prefix inference.

These policies generate chunks of future actions (e.g., 50 steps at a time) instead of single actions. Because the models are large, producing each chunk takes longer than the time it takes the robot to execute it. Naively executing chunks leads to problems such as pauses, jerky transitions, or sudden changes in strategy whenever the next chunk arrives late or disagrees with the previously executed actions.

RTC solves this by asynchronously generating the next chunk while the robot continues executing the current one, and by guiding the new chunk so it aligns smoothly with the portion of the previous chunk that has already been executed.

How RTC Works (simplified)

RTC lets the robot think ahead while it’s still moving. When the robot is carrying out one chunk of actions, RTC starts creating the next chunk early. But since the robot has already moved a bit by the time the new chunk is ready, RTC has to make sure the new chunk still lines up smoothly with what the robot is currently doing.

To do this, RTC treats the beginning of the new chunk like an inpainting or “fill-in-the-gaps” problem: it gently adjusts the first part of the new chunk so it blends naturally with the robot’s ongoing motion. The result is no pauses, no sudden jumps.

In technical terms, RTC adds a guidance term to the flow-matching denoising process that forces the overlapping timesteps of the new chunk to stay close to the executed portion of the previous chunk, typically using a soft transition mask.

Quick Start

Installation

RTC is built into LeRobot. Just install the policy dependencies you need:

# For Pi0 or Pi0.5
pip install -e ".[pi]"

# For SmolVLA
pip install -e ".[smolvla]"

Using RTC with Pi0

You can use lerobot-rollout --strategy.type=base --inference.type=rtc for RTC deployment on real robots. The snippet below provides a simplified pseudo-example of how RTC operates with Pi0 in your pipeline:

from lerobot.policies.pi0 import PI0Policy, PI0Config
from lerobot.configs import RTCAttentionSchedule
from lerobot.policies.rtc import RTCConfig, ActionQueue

# Load the Pi0 config so RTC can be enabled before the policy is built
policy_cfg = PI0Config.from_pretrained("lerobot/pi0_base")

# Enable RTC
policy_cfg.rtc_config = RTCConfig(
    enabled=True,
    execution_horizon=10,  # How many steps to blend with previous chunk
    max_guidance_weight=10.0,  # How strongly to enforce consistency
    prefix_attention_schedule=RTCAttentionSchedule.EXP,  # Exponential blend
)

# Load the policy
policy = PI0Policy.from_pretrained("lerobot/pi0_base", config=policy_cfg).to("cuda")

# Now use predict_action_chunk with RTC parameters
inference_delay = 4  # How many steps of inference latency, this value should be calculated based on the inference latency of the policy

# Initialize the action queue
action_queue = ActionQueue(policy_cfg.rtc_config)

# Start in a separate thread with the following function
def get_actions():
  while True:
    if should_get_actions:

      prev_actions = action_queue.get_left_over()
      obs = get_robot_observations(robot)

      # Generate actions WITH RTC
      actions = policy.predict_action_chunk(
          obs,
          inference_delay=inference_delay,
          prev_chunk_left_over=prev_actions,
      )

      action_queue.merge(
          actions, actions, inference_delay
      )

for step in range(num_steps):
    action = action_queue.get()

    # Execute the first N actions
    execute_actions(action)

Key Parameters

RTCConfig has the following parameters to tune:

mode selects the action-prefix conditioning method:

• guided (default) applies the original Jacobian guidance during denoising and works with ordinary flow-matching checkpoints.
• trained hard-inpaints the previous chunk’s prefix with per-action flow timesteps. It currently requires a Pi05 checkpoint trained with policy.rtc_training_max_delay > 0 and avoids the guidance backward pass.

Local rollout and the policy server apply the same conservative trained-mode admission rule: rtc_training_max_delay <= execution_horizon <= prediction_steps - rtc_training_max_delay. For Pi05, prediction_steps is the checkpoint’s chunk_size. This preserves the existing local limit: the configured prefix covers the maximum trained delay and leaves at least that many predicted steps outside it. Local rollout additionally requires inference.queue_threshold >= rtc_training_max_delay before connecting the robot; remote request timing uses its own refill settings.

Here execution_horizon is supplied prefix capacity, not the amount of playback between requests. Pi05 hard-conditions only the inference_delay prefix. The admission rule is not a latency guarantee or a claim that larger capacities must fail at runtime; measured overlap, remaining actions and freshness checks still determine whether a returned chunk can execute. Plain chunks and guided RTC are unaffected by this trained-only rule.

execution_horizon: How many timesteps from the previous chunk to maintain consistency with. Higher values mean smoother transitions but potentially less reactivity.

Typical values: 8-12 steps

RTCConfig(execution_horizon=10)

max_guidance_weight: How strongly to enforce consistency with the previous chunk. This is a hyperparameter that can be tuned to balance the smoothness of the transitions and the reactivity of the policy. For 10 steps flow matching (SmolVLA, Pi0, Pi0.5), a value of 10.0 is an optimal value.

prefix_attention_schedule: How to weight consistency across the overlap region.

• LINEAR: Linear decay from inference_delay to execution_horizon
• EXP: Exponential decay (recommended for getting started)
• ONES: Full weight across entire execution_horizon
• ZEROS: Binary (full weight up to inference_delay, then zero)

inference_delay: How many timesteps of inference latency your system has. This is passed to predict_action_chunk() rather than the config, since it may vary at runtime.

Testing RTC Offline

Before running on a real robot, test RTC with dataset samples to visualize how it works:

python examples/rtc/eval_dataset.py \
    --policy.path=lerobot/pi0_libero_finetuned \
    --dataset.repo_id=HuggingFaceVLA/libero \
    --rtc.execution_horizon=10 \
    --rtc.max_guidance_weight=10.0 \
    --device=cuda

Add --rtc.mode=trained when evaluating a compatible training-time RTC Pi05 checkpoint. Unsupported policies reject trained mode instead of falling back to guided RTC.

The script generates a visualization of the denoising process, comparing standard generation (left) with RTC (right). In the RTC plots, you can see how the first few steps (blue/purple lines) are guided to match the red ground truth trajectory (previous chunk’s tail), ensuring a smooth transition between chunks.

Testing RTC with a Real Robot

lerobot-rollout \
    --strategy.type=base \
    --policy.path=${HF_USERNAME}/policy_repo_id \
    --inference.type=rtc \
    --inference.rtc.mode=guided \
    --inference.rtc.execution_horizon=10 \
    --inference.rtc.max_guidance_weight=10.0 \
    --robot.type=so100_follower \
    --robot.port=/dev/tty.usbmodem58FA0834591 \
    --robot.cameras="{ gripper: {type: opencv, index_or_path: 1, width: 640, height: 480, fps: 30}, front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
    --task="Move green small object into the purple platform" \
    --duration=120 \
    --device=cuda

For a training-time RTC Pi05 checkpoint, change the mode to trained. The checkpoint records its maximum supported delay, and rollout validates measured latency against it:

lerobot-rollout \
    --strategy.type=base \
    --policy.path=${HF_USERNAME}/pi05_training_rtc \
    --inference.type=rtc \
    --inference.rtc.mode=trained \
    --inference.rtc.execution_horizon=10 \
    --robot.type=so100_follower \
    --robot.port=/dev/tty.usbmodem58FA0834591 \
    --task="Move green small object into the purple platform" \
    --duration=120 \
    --device=cuda

How RTC relates to remote inference

--inference.type=rtc is local RTC: inference runs in a background thread in the rollout process and uses no transport. Running a policy server and a robot client as separate processes on the same machine uses the remote backend instead.

Both asynchronous backends use the same bounded waiting and shutdown behavior on supported robots. See local RTC budgets for its timeout defaults. Waiting does not relax RTC’s prefix, delay or freshness checks.

Both RTC and remote inference improve real-time robot control, but they solve different problems.

Remote inference can use RTC when the checkpoint supports the negotiated RTC mode. Plain chunks also support optional cursor alignment and blending; see the remote inference guide for modes, compatibility and tuning.

Advanced: Debug Tracking

RTC includes built-in debug tracking to help you understand what’s happening during inference:

# Enable debug tracking
policy_cfg.rtc_config.debug = True
policy_cfg.rtc_config.debug_maxlen = 100
policy.init_rtc_processor()  # re-create the RTC processor so the new debug settings take effect

# After inference, access debug data
debug_data = policy.rtc_processor.get_all_debug_steps()

# Visualize denoising steps, corrections, etc.
from lerobot.policies.rtc.debug_visualizer import RTCDebugVisualizer
visualizer = RTCDebugVisualizer()
# ... create plots

See examples/rtc/eval_dataset.py for a complete example of offline RTC visualization.

References

• Smooth-As-Butter Robot Policies - Excellent technical explanation with real robot results
• Physical Intelligence - Real-Time Chunking - Original paper and research
• Kinetix RTC Implementation - Reference implementation from Physical Intelligence
• Training-Time Action Conditioning - Efficient RTC with clean-prefix conditioning during training
• RLDX-1 - PyTorch reference used for the training-time RTC integration