# LIBERO-plus

  The commands below run the simulator locally. To keep its dependencies away
  from your policy's, serve it instead with `lerobot-env-server
  --config_path=docker/sim/configs/libero_plus.yaml --sim.task= --docker=true`, where ``
  is the command's `--env.task`, and add `--env.server.endpoint=tcp/127.0.0.1:7448` to
  the command. A server runs the task it was started with (see [Simulation: local or served](./simulation)).

LIBERO-plus is a **robustness benchmark** for Vision-Language-Action (VLA) models built on top of [LIBERO](./libero). It systematically stress-tests policies by applying **seven independent perturbation dimensions** to the original LIBERO task set, exposing failure modes that standard benchmarks miss.

- Paper: [In-depth Robustness Analysis of Vision-Language-Action Models](https://arxiv.org/abs/2510.13626)
- GitHub: [sylvestf/LIBERO-plus](https://github.com/sylvestf/LIBERO-plus)
- Dataset: [lerobot/libero_plus](https://huggingface.co/datasets/lerobot/libero_plus)

![An overview of the LIBERO-plus benchmark perturbation dimensions](https://github.com/sylvestf/LIBERO-plus/raw/main/static/images/libero-plus.jpg)

## Perturbation dimensions

LIBERO-plus creates ~10 000 task variants by perturbing each original LIBERO task along these axes:

| Dimension             | What changes                                          |
| --------------------- | ----------------------------------------------------- |
| Objects layout        | Target position, presence of confounding objects      |
| Camera viewpoints     | Camera position, orientation, field-of-view           |
| Robot initial states  | Manipulator start pose                                |
| Language instructions | LLM-rewritten task description (paraphrase / synonym) |
| Light conditions      | Intensity, direction, color, shadow                   |
| Background textures   | Scene surface and object appearance                   |
| Sensor noise          | Photometric distortions and image degradation         |

## Available task suites

LIBERO-plus perturbs the 40 original tasks in the Spatial, Object, Goal, and Long suites. Each registered task in these four suites is a perturbed variant, rather than one of the original tasks. The fork also retains the 90 unperturbed tasks in LIBERO-90.

| Suite          | CLI name         | Original tasks | Registered tasks | Max steps | Description                                       |
| -------------- | ---------------- | -------------- | ---------------- | --------- | ------------------------------------------------- |
| LIBERO-Spatial | `libero_spatial` | 10             | 2,402            | 280       | Tasks requiring reasoning about spatial relations |
| LIBERO-Object  | `libero_object`  | 10             | 2,518            | 280       | Tasks centered on manipulating different objects  |
| LIBERO-Goal    | `libero_goal`    | 10             | 2,591            | 300       | Goal-conditioned tasks with changing targets      |
| LIBERO-90      | `libero_90`      | 90             | 90               | 400       | Unperturbed short-horizon LIBERO tasks            |
| LIBERO-Long    | `libero_10`      | 10             | 2,519            | 520       | Long-horizon tasks from the LIBERO-100 collection |

These counts come from the upstream [task registry](https://github.com/sylvestf/LIBERO-plus/blob/4976dc30028e805ff8094b55501d532c48fec182/libero/libero/benchmark/libero_suite_task_map.py). The four perturbed suites contain **10,030 task variants** in total; LIBERO-90 is not included in that total. Counts and task indices depend on the installed fork revision.

  Installing LIBERO-plus **replaces** vanilla LIBERO — it uninstalls `hf-libero`
  so that `import libero` resolves to the LIBERO-plus fork. You cannot have both
  installed at the same time. To switch back to vanilla LIBERO, uninstall the
  fork and reinstall with `pip install -e ".[libero]"`.

## Installation

### System dependencies (Linux only)

```bash
sudo apt install libexpat1 libfontconfig1-dev libmagickwand-dev
```

### Python package

```bash
pip install -e ".[libero]" "robosuite==1.4.1" bddl easydict mujoco wand scikit-image gym
git clone https://github.com/sylvestf/LIBERO-plus.git
cd LIBERO-plus && pip install --no-deps -e .
pip uninstall -y hf-libero  # so `import libero` resolves to the fork
```

LIBERO-plus is installed from its GitHub fork rather than a pyproject extra — the fork ships as a namespace package that pip can't handle, so it must be cloned and added to `PYTHONPATH`. See the `libero_plus_native` stage of `docker/Dockerfile.sim` for the canonical install. MuJoCo is required, so only Linux is supported.

Set the MuJoCo rendering backend before running evaluation:

```bash
export MUJOCO_GL=egl   # headless / HPC / cloud
```

### Download LIBERO-plus assets

LIBERO-plus ships its extended asset pack separately. Download `assets.zip` from the [Hugging Face dataset](https://huggingface.co/datasets/Sylvest/LIBERO-plus/tree/main) and extract it into the LIBERO-plus package directory:

```bash
# After installing the package, find where it was installed:
python -c "import libero; print(libero.__file__)"
# Then extract assets.zip into <package_root>/libero/assets/
```

## Evaluation

### Default local evaluation

Evaluate every registered variant across the four perturbed suites, using one episode per variant as recommended by the [upstream evaluation guide](https://github.com/sylvestf/LIBERO-plus#-evaluation):

```bash
lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
  --eval.batch_size=1 \
  --eval.n_episodes=1 \
  --env.max_parallel_tasks=1
```

This runs **10,030 episodes** for the registry above. With `--eval.n_episodes=10`, the same command runs **100,300 episodes**, not 400.

### Smoke test

Before launching a full evaluation, check your setup with one registered variant and one episode:

```bash
lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial \
  --env.task_ids='[0]' \
  --eval.batch_size=1 \
  --eval.n_episodes=1
```

This is a setup check, not a representative robustness evaluation.

### Single-suite evaluation

Evaluate on one LIBERO-plus suite:

```bash
lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial \
  --eval.batch_size=1 \
  --eval.n_episodes=1
```

- `--env.task` picks the suite (`libero_spatial`, `libero_object`, etc.).
- `--env.task_ids` restricts to registered task indices (`[0]`, `[1,2,3]`, etc.), which identify variants in the four perturbed suites. Omit to run all registered tasks. When evaluating multiple suites, the same index list is applied independently to each suite.
- `--eval.batch_size` controls how many environments run in parallel.
- `--eval.n_episodes` sets how many episodes to run per registered task. Increasing it repeats evaluation of the selected variants; it does not select additional distinct variants.

### Multi-suite evaluation

Benchmark a policy across multiple suites at once by passing a comma-separated list:

```bash
lerobot-eval \
  --policy.path="your-policy-id" \
  --env.type=libero_plus \
  --env.task=libero_spatial,libero_object \
  --eval.batch_size=1 \
  --eval.n_episodes=1
```

### Control mode

LIBERO-plus supports two control modes — `relative` (default) and `absolute`. Different VLA checkpoints are trained with different action parameterizations, so make sure the mode matches your policy:

```bash
--env.control_mode=relative   # or "absolute"
```

### Reset performance

By default, LeRobot preserves LIBERO's hard-reset behavior. With fixed initial
states enabled, you can opt into soft resets to skip rebuilding the simulator
model and renderer on every episode:

```bash
--env.init_states=true --env.hard_reset=false
```

Soft resets are faster but are not bit-identical to hard resets after the
environment's settling steps, so camera observations and policy results may
differ slightly. Use hard resets when reproducing benchmark results.

### Policy inputs and outputs

**Observations:**

- `observation.state` — 8-dim proprioceptive features (eef position, axis-angle orientation, gripper qpos)
- `observation.images.image` — main camera view (`agentview_image`), HWC uint8
- `observation.images.image2` — wrist camera view (`robot0_eye_in_hand_image`), HWC uint8

**Actions:**

- Continuous control in `Box(-1, 1, shape=(7,))` — 6D end-effector delta + 1D gripper

### Evaluation workload and reproducibility

The total number of evaluation episodes is **the number of selected registered tasks across all suites × `--eval.n_episodes`**. For example, all Spatial variants with one episode each give 2,402 episodes; selecting `--env.task_ids='[0,1,2]'` in Spatial and Object with one episode each gives six episodes.

The 400-episode calculation (40 original tasks × 10 episodes) applies to vanilla LIBERO, not a full LIBERO-plus evaluation. For LIBERO-plus, start with one episode per registered variant. Repeated episodes of a variant are not equivalent to sampling distinct variants of an original task.

For a custom subset, use the fork's [task classification](https://github.com/sylvestf/LIBERO-plus/blob/4976dc30028e805ff8094b55501d532c48fec182/libero/libero/benchmark/task_classification.json) to identify perturbation categories and difficulty levels. Do not assume that every original task has ten distinct variants in every perturbation category. Record the fork revision, selected suite names and task indices, episodes per variant, seed, and reset settings when reporting results. To compare with published results, follow the benchmark's sampling and aggregation protocol rather than treating an arbitrary subset as equivalent.

## Training

### Dataset

A LeRobot-format training dataset for LIBERO-plus is available at:

- [lerobot/libero_plus](https://huggingface.co/datasets/lerobot/libero_plus)

### Example training command

```bash
lerobot-train \
    --policy.type=smolvla \
    --policy.repo_id=${HF_USER}/smolvla_libero_plus \
    --policy.load_vlm_weights=true \
    --dataset.repo_id=lerobot/libero_plus \
    --env.type=libero_plus \
    --env.task=libero_spatial \
    --output_dir=./outputs/ \
    --steps=100000 \
    --batch_size=4 \
    --eval.batch_size=1 \
    --eval.n_episodes=1 \
    --env_eval_freq=1000
```

## Relationship to LIBERO

LIBERO-plus is a drop-in extension of LIBERO:

- Same Python gym interface (`LiberoEnv`) and canonical observations
- Same camera names and observation/action format
- Same task suite names
- Installs under the same `libero` Python package name (different GitHub repo)

To use the original LIBERO benchmark, see [LIBERO](./libero) and use `--env.type=libero`.