Extensive Language Support in LeRobot Dataset

Tell a robot “put the kettle on the stove and turn it on,” then change your mind halfway through: “actually, use the front burner.” A useful robot should update its plan, remember that it already filled the kettle, answer questions about what it sees, and occasionally talk back. None of that fits in one task label.

Until now, a LeRobot dataset has known exactly one sentence about each episode: its task. Richer behavior needs language throughout the trajectory: the plan, active subtask, memory, visual questions and answers, interjections, and tool calls.

This article follows that language from raw robot video to model-ready training examples:

• Two optional LeRobotDataset columns store persistent state and frame-level events without breaking existing datasets.
• The automatic annotation pipeline (lerobot-annotate) watches each episode with a vision-language model and fills those columns.
• Recipes select annotations in time and render policy-agnostic chat messages for different training objectives.
• Tool schemas and invocations let datasets supervise structured actions such as say(...).

These interfaces are our best effort to consolidate how task annotation and language training work across the field. They may not be optimal for every future policy. The priority is a flexible boundary: annotation records what happened and when, recipes define what to learn, and policies remain free to choose how those messages become tokens and losses.

Everything here is open source and shipping in LeRobot. Docs: language and recipes, annotation pipeline, tools.

The article reads in data-flow order: storage → automatic annotation → recipes and tools. Each section also stands on its own if you want to jump directly to the format, pipeline, or training objective you need.

Read https://lerobot-robots-that-talk.hf.space/llm.txt, explain how language is stored, annotated, and turned into policy-agnostic training messages, then answer my question with intuition first and specific evidence from the article; distinguish shipped features from experiments and future work, define technical terms, and state when the evidence is insufficient. My question:

Language in the dataset format

First, storage: where does this language live, and how do we add it without breaking existing datasets?

The answer is two optional columns next to the frame data in data/chunk-*/file-*.parquet. Datasets without them behave exactly as before; datasets with them gain a full language track. The split between the two columns is the most important decision in the format, so it is worth understanding first.

Collapsing these into one column would force you to repeat events on every frame, which grows to a massive amount of data. Storing it per episode would require multiple reads or seeks when reading random frames and would increase data-loading cost. Splitting them makes both cheap: persistent rows dictionary-encode in parquet to nearly nothing because the same row repeats, and event rows only exist where something fired, so storage scales with emissions.

The row schema

Both columns share the same fields except for time: persistent rows carry the timestamp at which they became active, while event rows inherit the timestamp of the frame on which they are stored:

role: string
content: string | null
style: string | null
timestamp: float32        # persistent rows only
camera: string | null     # observation.images.* key, for view-dependent rows
tool_calls: list[Json] | null

The camera field tags rows whose content is grounded in a specific view. View-dependent styles (vqa, trace) must set it to the matching observation.images.*

The styles that live in each column

Annotations come in named styles:

Explore an annotated episode

The new LeRobot Dataset Visualizer renders these language tracks alongside the camera streams and robot state. Scrub the timeline, inspect each annotation, and see how persistent subtasks line up with the episode:

So the dataset stores rows of language tagged by style, role, frame timing, and camera where needed. By itself that is just a format. The next question is how those rows get there: the annotation pipeline turns raw episodes into subtasks, plans, memory, interjections, and VQA automatically.

The annotation pipeline

Labeling demonstrations with subtasks is not a new idea. What changed is that it finally became cheap and reliable enough to do automatically, at the scale a dataset needs. Hand-annotating is brutal: a 30-second episode at 30 Hz is 900 frames, and marking subtask boundaries, writing a plan, and deciding what to remember, consistently across thousands of episodes, is hours of tedious work. Two annotators don’t always agree either.

The obvious shortcut, “ask a model to do it”, did not work well enough for a long time: VLMs hallucinated objects and cut boundaries inconsistently, and annotations you cannot trust are worse than none because you train on them. That is what flipped. Current open VLMs are finally good enough. Self-hosted behind an OpenAI-compatible endpoint, they can watch an episode and produce subtask boundaries, plans, and grounded answers, not perfectly but well enough to leverage. The shipped path uses Qwen with vllm serve on a Hugging Face Job.

Concretely, pointed at a 30-second “make a cup of tea” episode, the pipeline returns something like this, with timestamps read straight off the frames:

0.0–6.2s   go to the counter
6.2–13.0s  pick up the kettle
13.0–21.4s place the kettle on the front burner
21.4–27.0s turn the burner on
plan:      1. pick up kettle  2. place on burner  3. turn it on
memory @13s: I picked up the kettle.
vqa @8s:   Q: where is the kettle?  A: [bbox 412, 260, 488, 360]

A year ago the same prompt would have invented a teapot that was never in frame and split one grab across two boundaries. The difference is not a clever trick; the base models got good enough to ground.

That is what lerobot-annotate does: point it at a LeRobot dataset with no language and it watches each episode with a VLM, writing subtasks, plans, memory, interjections, and VQA into both language columns straight in data/chunk-*/file-*.parquet.

How subtasks are generated

The plan module does not ask for subtasks in one shot. It uses the two-step describe then segment flow:

1. Describe. The VLM narrates only what it actually sees in the chosen camera, with no guessing about the task.
2. Segment. That grounded description is fed back in, and the VLM splits the episode into consecutive atomic subtasks, guided by a causal event-boundary rule (a new event starts when an object becomes held, is released, reaches a new location, or a lid changes state).

Both passes see the episode as timestamped contact sheets: frames sampled at a fixed rate (one every 0.5s by default) packed into JPEG grids, each frame’s time burned into its corner so the model can cite exact boundary times. This is far cheaper in vision tokens than one image per frame, so sampling stays dense; long episodes are windowed at the same density and merged.

Running it on Hugging Face Jobs

Annotating a real dataset needs a GPU large enough to serve the VLM, so lerobot-annotate can dispatch itself to Hugging Face Jobs. Authenticate once, then add --job.target=<flavor> to the same command you would run locally:

hf auth login

uv run lerobot-annotate \
    --repo_id=user/my_dataset \
    --new_repo_id=user/my_dataset_annotated \
    --push_to_hub=true \
    --vlm.model_id=Qwen/Qwen3.6-27B \
    --vlm.num_gpus=1 \
    --vlm.serve_command="vllm serve Qwen/Qwen3.6-27B --tensor-parallel-size 1 --max-model-len 32768 --gpu-memory-utilization 0.8 --uvicorn-log-level warning --port {port}" \
    --vlm.serve_ready_timeout_s=1800 \
    --vlm.chat_template_kwargs='{"enable_thinking": false}' \
    --job.target=h200

This submits a single-GPU h200 job, installs LeRobot on the vllm/vllm-openai image, boots the VLM server, and runs the annotation modules across the dataset. The command streams the remote logs; Ctrl-C detaches without cancelling the job. With --push_to_hub=true, the result is uploaded to --new_repo_id (or back to --repo_id when that is unset). Without it, the annotated dataset disappears when the job ends.

For a larger dataset, switch to --job.target=h200x4, set --vlm.parallel_servers and --vlm.num_gpus to match, and increase --job.timeout. Changing only the hardware flavor still starts one VLM server.

The full flag reference and validation rules live in the annotation-pipeline docs.

Roughly what it costs

A concrete number: annotating 100 typical 30-second, single-camera episodes costs roughly $1.20 of H200 compute in our runs. For small datasets, much of the job is provisioning the worker and loading the model into vLLM rather than processing episodes. Reusing a warm server or batching more episodes into one job can reduce the effective cost per episode further.

Benchmarking subtasks on WGO-Bench

To benchmark the subtask slice, we ran lerobot-annotate (Qwen3.6-27B, self-hosted vLLM) on WGO-Bench, Macrodata Labs’ public benchmark of 100 robot and egocentric episodes with 743 human-annotated subtask segments. It scores 0.197 Segment F1, 0.610 matched-label accuracy, and 0.121 end-to-end semantic F1, roughly 1.8× the 0.110 open baseline (Qwen 3.6 Flash).

Are those results good? They are an encouraging first result: the end-to-end score is roughly 1.8× the open baseline, and once a predicted segment matches a human segment, its label agrees 61% of the time. The clearest opportunity is boundary detection—the 0.197 Segment F1 shows that predicted transitions do not yet consistently align with human segmentation. Because the stricter 0.121 end-to-end score combines both boundary and label quality, the pipeline is already useful for bootstrapping annotations at scale while leaving substantial room for improvement. Credit to Macrodata for the benchmark, annotation protocol, and segmentation recipes we incorporated.

Reproduce the run. Benchmark dataset: macrodata/WGO-Bench. Annotated output (full 100 episodes): pepijn223/wgo_bench_lerobot_subtask_qwen_full.

Where the ideas come from

The pipeline stands on prior work, roughly one influence per module:

The pipeline is a first step for automatic language annotations. It is built to grow. A new module (trajectory traces, affordances), a new prompt template, or a quality fix to an existing module is a very welcome PR. Modules live under src/lerobot/annotations/steerable_pipeline/modules/. Once those rows exist, the next chapter shows how recipes select them in time and turn them into training samples.

Recipes: from rows to training samples

Across robot learning, task labels, step annotations, memory targets, and high-level reasoning are often represented differently for each dataset and policy. This format is our best effort to consolidate those patterns into one path from annotation to language training. We do not expect every decision here to be optimal for every future policy. We prioritize a stable, flexible boundary: the dataset records what happened and when, a recipe defines what to learn from it, and each policy decides how to serialize and train on the resulting messages.

That separation keeps data annotation, System 1 training, and System 2 training from becoming one fixed pipeline. You can revise an annotation module without redesigning the policy, change a training objective without rewriting the dataset, or reuse the same rendered messages with a different model processor.

Storing language is therefore only half the problem. At training time, we must decide which rows a sample sees and how they become a conversation. A memory-update objective needs the previous memory and completed subtask; VQA needs a question, image, and answer; low-level control needs the task and active subtask. Same dataset, different samples. A recipe expresses that choice as YAML config, and the renderer emits policy-agnostic messages rather than applying a tokenizer template.

Anatomy of a recipe

Recipes are built from a few small primitives:

1. An annotation row stores a value such as a subtask, memory, or VQA event.
2. A resolver selects a row relative to the sampled frame t.
3. A binding gives that selected value a local name.
4. A message inserts bound values into a user or assistant turn.
5. target: true marks a turn for supervised generation by the policy.
6. stream labels a turn as high_level or low_level metadata; the policy processor decides what that label means.

The stream label does not select a separate model or loss by itself. PI052 interprets a target as language-generation supervision and any low_level turn as enabling action-flow supervision, so one sample can train language, actions, or both through the same shared VLM. Here is a joint text-and-action objective:

bindings:
  active_subtask: "active_at(t, style=subtask)"
messages:
  - { role: user, content: "${task}", stream: high_level }
  - { role: assistant, content: "${active_subtask}", stream: low_level, target: true }

For one sampled frame, the renderer runs active_at(...), stores the result as active_subtask, substitutes it into the assistant message, and records that message as a target. PI052 uses the target for language loss and the low_level label for action-flow loss. A recipe containing only targeted high_level turns trains language generation without action flow. ${task} is a built-in binding: it uses a deterministic task_aug rephrasing when available and otherwise falls back to the canonical task string.

Bindings and temporal resolvers

Resolvers describe which value to select, not how that value is tokenized:

• active_at(t, style=subtask): the persistent subtask row in effect at time t.
• nth_prev(style=memory, offset=1): the previous emitted memory, not the previous frame.
• nth_next(style=subtask, offset=1): the next subtask transition after the active one.
• emitted_at(t, style=interjection): an event row on exactly this frame.

Blends: one dataset, many objectives

A recipe can branch into a weighted blend of sub-recipes. On ordinary frames, the sample index deterministically selects one branch according to those weights, so the same index always gets the same objective. Frames carrying sparse VQA annotations are the exception: a matching ask_vqa* branch takes priority over the weighted draw so the label is not wasted.

View-dependent resolution for VQA

With multiple cameras, a (vqa, user)/(vqa, assistant) pair can fire per camera at the same timestamp. A camera= filter (parallel to role=) disambiguates, with one sub-recipe and image block per camera:

ask_vqa_top:
  bindings:
    vqa_query: "emitted_at(t, style=vqa, role=user, camera=observation.images.top)"
    vqa: "emitted_at(t, style=vqa, role=assistant, camera=observation.images.top)"
  messages:
    - role: user
      stream: high_level
      if_present: vqa_query
      content:
        - { type: image, feature: observation.images.top }
        - { type: text, text: "${vqa_query}" }
    - { role: assistant, content: "${vqa}", stream: high_level, target: true, if_present: vqa }

Tools: when the robot calls a function

Sometimes the right action is not a motor command but to say something, capture an extra observation, or log an event. LeRobot models this the way LLMs do: with tool calls. An assistant turn emits a structured invocation like say(text="On it."); the dataset stores it and the policy is trained to generate it. Actually dispatching a generated call to a real implementation at inference is a separate layer we return to below.

Tools split into two layers: the catalog (what can be called) and the invocations (an actual call, with arguments, on a specific frame).

The catalog

The catalog is a list of OpenAI-style function schemas, stored at meta/info.json["tools"]:

{
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "say",
        "description": "Speak a short utterance to the user via the TTS executor.",
        "parameters": {
          "type": "object",
          "properties": {
            "text": { "type": "string", "description": "The verbatim text to speak." }
          },
          "required": ["text"]
        }
      }
    }
  ]
}

You read it through the metadata accessor, which falls back to DEFAULT_TOOLS (currently the single canonical say schema) when a dataset declares none, so chat-template consumers keep working with no configuration:

from lerobot.datasets.dataset_metadata import LeRobotDatasetMetadata

meta = LeRobotDatasetMetadata(repo_id="pepijn/super_poulain_final_annotations")
tools = meta.tools     # list[dict] of OpenAI tool schemas, DEFAULT_TOOLS if none declared

prompt_str = tokenizer.apply_chat_template(sample["messages"], tools=meta.tools, tokenize=False)
print(prompt_str)

The final line shows the exact serialized prompt the tokenizer receives.

The invocation, stored per row

The actual call is stored per frame, on assistant atoms in language_events:

{
  "role": "assistant",
  "content": null,
  "style": null,
  "camera": null,
  "tool_calls": [
    { "type": "function",
      "function": { "name": "say", "arguments": { "text": "On it." } } }
  ]
}

Recipes splice these into a rendered turn with tool_calls_from. The shipped speech recipe creates a tool-call-only assistant target:

user_interjection_response:
  bindings:
    interjection: "emitted_at(t, style=interjection)"
    speech: "emitted_at(t, role=assistant, tool_name=say)"
  messages:
    - { role: user, content: "${task}", stream: high_level }
    - { role: user, content: "${interjection}", stream: high_level, if_present: interjection }
    - { role: assistant, stream: high_level, target: true, if_present: speech, tool_calls_from: speech }

The renderer attaches the structured call to the message. PI052’s processor serializes it as a <say>...</say> text target, and its runtime can parse that marker back out.

Runtime execution is not shipped yet: LeRobot does not yet include a tool registry or TTS dispatcher, so parsed say text is not spoken. For now, declare a schema to train generation; implementing and dispatching the call is an open extension described in the tool-calling docs.

Part 2 preview: policies in action

This post stops at the policy-agnostic training example. Part 2: Language-Conditioned Robot Policies, coming later, follows that example into PI052: how System 2 generates language, how System 1 turns a subtask into actions, how the runtime connects them, and what happens in RoboCasa.

Here is a preview of the simulation experiments:

Get involved

This release is a foundation, not a finish line. The dataset format, annotation pipeline, and recipes were built to be extended, and the most useful next language style or training objective may come from your data.

Three ways to contribute

1. Annotate a dataset. Grab an existing LeRobot dataset, run lerobot-annotate through HF Jobs, and inspect the generated subtasks, plans, memory, and VQA in .annotate_staging/ and the two language columns. Share the annotated dataset and what the pipeline got wrong.

2. Write a recipe. Combine the same annotations into a new objective, inspect the rendered messages, and share the YAML. Recipes deliberately stay independent of tokenizers and model implementations.

3. Add a pipeline module or tool. Add trajectory traces, affordances, grasp points, a sharper grounding flow, or a prompt-quality fix to plan / interjections / vqa. The tool-execution layer that turns a generated say into real speech is also still open.

Why this matters

Robotics is deeply multimodal, but model progress depends on data that makes the right intermediate concepts visible. Putting language beside robot trajectories gives researchers a common substrate for testing how subtasks, memory, grounded answers, and interaction affect learning.

So annotate something, train something, break something, and tell us all about it.

Pepijn Kooijmans @pepijn223

Steven Palma @imstevenpmwork

Caroline Pascal @CarolinePascal

Jade Choghari @jadechoghari

Khalil Meftah @lilkm

Martino Russi @nepyope

Nicolas Rabault @Nico-robot

Nikodem Bartnik @nikodembartnik

Leandro von Werra @lvwerra

Thomas Wolf @thomwolf

Code and docs

• Language columns and recipes docs
• Annotation pipeline docs
• Tools docs

Papers

• π0.5 Black et al. (2025). π0.5: A Vision-Language-Action Model with Open-World Generalization. arxiv.org/abs/2504.16054
• Hi Robot Shi et al. (2025). Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models. arxiv.org/abs/2502.19417
• Pi0.7 Physical Intelligence (2025). π0.7. pi.website/pi07
• MEM Torne et al. (2026). Memory for Vision-Language-Action Models. arxiv.org/abs/2603.03596
• ECoT Zawalski et al. (2024). Robotic Control via Embodied Chain-of-Thought Reasoning. arxiv.org/abs/2407.08693
• Steerable VLA Policies Chen et al. (2026). Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control. arxiv.org/abs/2602.13193