# Extensive Language Support in LeRobot Dataset Agent-readable edition This file contains the complete article in reading order. Interactive figures are represented by their titles, descriptions, and links. Preserve technical names and commands exactly when discussing the article. Recommended explanation style: Lead with intuition, then make it concrete with small worked numbers and examples. Be visual. Use ASCII diagrams, side-by-side comparisons, and shared notation so differences are easy to see. Render math as terminal-readable text using forms such as 1/√L, Σ, and α instead of LaTeX delimiters. Define every symbol and metric. Unpack compressed ideas instead of assuming them. Keep prose concise and avoid filler. Canonical article: https://huggingface.co/spaces/lerobot/robots-that-talk Tell a robot "put the kettle on the stove and turn it on," then change your mind halfway through: "actually, use the front burner." A useful robot should update its plan, remember that it already filled the kettle, answer questions about what it sees, and occasionally talk back. None of that fits in one task label. Until now, a LeRobot dataset has known exactly one sentence about each episode: its `task`. Richer behavior needs **language throughout the trajectory**: the plan, active subtask, memory, visual questions and answers, interjections, and tool calls. This article follows that language from raw robot video to model-ready training examples: - Two optional **LeRobotDataset columns** store persistent state and frame-level events without breaking existing datasets. - The automatic **annotation pipeline** (`lerobot-annotate`) watches each episode with a vision-language model and fills those columns. - **Recipes** select annotations in time and render policy-agnostic chat messages for different training objectives. - **Tool schemas and invocations** let datasets supervise structured actions such as `say(...)`. These interfaces are our best effort to consolidate how task annotation and language training work across the field. They may not be optimal for every future policy. The priority is a flexible boundary: annotation records what happened and when, recipes define what to learn, and policies remain free to choose how those messages become tokens and losses. -------- Everything here is open source and shipping in LeRobot. **Docs**: [language and recipes](https://huggingface.co/docs/lerobot/main/en/language_and_recipes), [annotation pipeline](https://huggingface.co/docs/lerobot/main/en/annotation_pipeline), [tools](https://huggingface.co/docs/lerobot/main/en/tools). The article reads in data-flow order: storage → automatic annotation → recipes and tools. Each section also stands on its own if you want to jump directly to the format, pipeline, or training objective you need. Read the interactive article here, or give the agent-readable version to your preferred assistant and explore it through questions. - **Interactive article:** read the full post and explore the figures. - **Agent-readable article:** use [llm.txt](/llm.txt), which contains the complete post in reading order with figure descriptions and links. Copy this prompt and add your question: ```text Read https://lerobot-robots-that-talk.hf.space/llm.txt, explain how language is stored, annotated, and turned into policy-agnostic training messages, then answer my question with intuition first and specific evidence from the article; distinguish shipped features from experiments and future work, define technical terms, and state when the evidence is insufficient. My question: ``` ## Language in the dataset format First, storage: where does this language live, and how do we add it without breaking existing datasets? The answer is **two optional columns** next to the frame data in `data/chunk-*/file-*.parquet`. Datasets without them behave exactly as before; datasets with them gain a full language track. The split between the two columns is the most important decision in the format, so it is worth understanding first. [Figure: Two language columns] Hover a style to see which column it lands in: language_persistent holds state that stays true until replaced, language_events holds things that happen on one exact frame. Collapsing these into one column would force you to repeat events on every frame, which grows to a massive amount of data. Storing it per episode would require multiple reads or seeks when reading random frames and would increase data-loading cost. Splitting them makes both cheap: persistent rows dictionary-encode in parquet to nearly nothing because the same row repeats, and event rows only exist where something fired, so storage scales with emissions. ### The row schema Both columns share the same fields except for time: persistent rows carry the timestamp at which they became active, while event rows inherit the timestamp of the frame on which they are stored: ```text role: string content: string | null style: string | null timestamp: float32 # persistent rows only camera: string | null # observation.images.* key, for view-dependent rows tool_calls: list[Json] | null ``` The `camera` field tags rows whose content is grounded in a specific view. View-dependent styles (`vqa`, `trace`) must set it to the matching `observation.images.*` meta/tasks.parquet stays the canonical home for the task string. When task_aug rows exist, the special ${task} binding deterministically rotates through those rephrasings by sample index; otherwise it falls back to the canonical task. An explicit task override always wins. ### The styles that live in each column Annotations come in named **styles**: | Style | Column | What it is | |:---|:---|:---| | `subtask` | `language_persistent` | The active atomic step, π0.7-style "how, not what" | | `plan` | `language_persistent` | The high-level plan | | `memory` | `language_persistent` | Compressed memory of what has happened (MEM-style) | | `motion` | `language_persistent` | A camera-independent motion primitive | | `task_aug` | `language_persistent` | Rephrasings of the task for augmentation | | `interjection` | `language_events` | A user interjection at a single instant | | `vqa` | `language_events` | A grounded user/assistant question-answer pair | | `trace` | `language_events` | A view-dependent pixel trajectory | [Figure: Language along one episode] Scrub the playhead: persistent lanes (plan, subtask, memory) always show their currently-active value, while event pins (VQA, interjection, speech) fire only on their exact frame. This is exactly what a recipe resolves against. ### Explore an annotated episode The new [LeRobot Dataset Visualizer](https://huggingface.co/spaces/lerobot/visualize_dataset?path=%2Fpepijn223%2Fsuper_chatton_annotated_full%2Fepisode_1%3Ftab%3Dannotations) renders these language tracks alongside the camera streams and robot state. Scrub the timeline, inspect each annotation, and see how persistent subtasks line up with the episode: [Interactive: LeRobot Dataset Visualizer showing language annotations] URL: https://lerobot-visualize-dataset.hf.space/?path=%2Fpepijn223%2Fsuper_chatton_annotated_full%2Fepisode_1%3Ftab%3Dannotations So the dataset stores rows of language tagged by style, role, frame timing, and camera where needed. By itself that is just a format. The next question is how those rows get there: the annotation pipeline turns raw episodes into subtasks, plans, memory, interjections, and VQA automatically. ## The annotation pipeline Labeling demonstrations with subtasks is not a new idea. What changed is that it finally became cheap and reliable enough to do automatically, at the scale a dataset needs. Hand-annotating is brutal: a 30-second episode at 30 Hz is 900 frames, and marking subtask boundaries, writing a plan, and deciding what to remember, consistently across thousands of episodes, is hours of tedious work. Two annotators don't always agree either. The obvious shortcut, "ask a model to do it", did not work well enough for a long time: VLMs hallucinated objects and cut boundaries inconsistently, and annotations you cannot trust are worse than none because you train on them. That is what flipped. **Current open VLMs are finally good enough.** Self-hosted behind an OpenAI-compatible endpoint, they can watch an episode and produce subtask boundaries, plans, and grounded answers, not perfectly but well enough to leverage. The shipped path uses Qwen with `vllm serve` on a Hugging Face Job. Concretely, pointed at a 30-second "make a cup of tea" episode, the pipeline returns something like this, with timestamps read straight off the frames: ```text 0.0–6.2s go to the counter 6.2–13.0s pick up the kettle 13.0–21.4s place the kettle on the front burner 21.4–27.0s turn the burner on plan: 1. pick up kettle 2. place on burner 3. turn it on memory @13s: I picked up the kettle. vqa @8s: Q: where is the kettle? A: [bbox 412, 260, 488, 360] ``` A year ago the same prompt would have invented a teapot that was never in frame and split one grab across two boundaries. The difference is not a clever trick; the base models got good enough to ground. That is what `lerobot-annotate` does: point it at a LeRobot dataset with no language and it watches each episode with a VLM, writing subtasks, plans, memory, interjections, and VQA into both language columns straight in `data/chunk-*/file-*.parquet`. [Figure: lerobot-annotate, end to end] One shared Qwen-VL server drives all three modules, each staging JSONL that the validator checks before the writer rewrites parquet and updates the tool metadata in meta/info.json. ### How subtasks are generated The `plan` module does not ask for subtasks in one shot. It uses the two-step **describe then segment** flow: 1. **Describe.** The VLM narrates only what it actually sees in the chosen camera, with no guessing about the task. 2. **Segment.** That grounded description is fed back in, and the VLM splits the episode into consecutive atomic subtasks, guided by a causal event-boundary rule (a new event starts when an object becomes held, is released, reaches a new location, or a lid changes state). Both passes see the episode as **timestamped contact sheets**: frames sampled at a fixed rate (one every 0.5s by default) packed into JPEG grids, each frame's time burned into its corner so the model can cite exact boundary times. This is far cheaper in vision tokens than one image per frame, so sampling stays dense; long episodes are windowed at the same density and merged. The contact-sheet format comes from [Macrodata Labs](https://macrodata.co/blog/annotating-robot-video-subtasks#contact-sheets-better-and-cheaper), building on [Scale's dense-captioning work](https://labs.scale.com/blog/path-to-large-scale-dense-video-captioning) (contact sheets beat per-frame vision tokens for long-horizon labeling). Macrodata's [field report](https://macrodata.co/blog/annotating-robot-video-subtasks#final-segmentation-recipe) added the atomic completed-event boundaries and cheap second-pass relabel that lerobot-annotate follows, on top of the π0.7-style describe-then-segment flow. ### Running it on Hugging Face Jobs Annotating a real dataset needs a GPU large enough to serve the VLM, so `lerobot-annotate` can dispatch itself to [Hugging Face Jobs](https://huggingface.co/docs/hub/en/jobs). Authenticate once, then add `--job.target=` to the same command you would run locally: ```bash hf auth login uv run lerobot-annotate \ --repo_id=user/my_dataset \ --new_repo_id=user/my_dataset_annotated \ --push_to_hub=true \ --vlm.model_id=Qwen/Qwen3.6-27B \ --vlm.num_gpus=1 \ --vlm.serve_command="vllm serve Qwen/Qwen3.6-27B --tensor-parallel-size 1 --max-model-len 32768 --gpu-memory-utilization 0.8 --uvicorn-log-level warning --port {port}" \ --vlm.serve_ready_timeout_s=1800 \ --vlm.chat_template_kwargs='{"enable_thinking": false}' \ --job.target=h200 ``` This submits a single-GPU `h200` job, installs LeRobot on the `vllm/vllm-openai` image, boots the VLM server, and runs the annotation modules across the dataset. The command streams the remote logs; `Ctrl-C` detaches without cancelling the job. With `--push_to_hub=true`, the result is uploaded to `--new_repo_id` (or back to `--repo_id` when that is unset). Without it, the annotated dataset disappears when the job ends. For a larger dataset, switch to `--job.target=h200x4`, set `--vlm.parallel_servers` and `--vlm.num_gpus` to match, and increase `--job.timeout`. Changing only the hardware flavor still starts one VLM server. The full flag reference and validation rules live in the [annotation-pipeline docs](https://huggingface.co/docs/lerobot/main/en/annotation_pipeline). ### Roughly what it costs **A concrete number:** annotating 100 typical 30-second, single-camera episodes costs roughly **$1.20 of H200 compute** in our runs. For small datasets, much of the job is provisioning the worker and loading the model into vLLM rather than processing episodes. Reusing a warm server or batching more episodes into one job can reduce the effective cost per episode further. ### Benchmarking subtasks on WGO-Bench To benchmark the **subtask** slice, we ran `lerobot-annotate` (Qwen3.6-27B, self-hosted vLLM) on [WGO-Bench](https://huggingface.co/datasets/macrodata/WGO-Bench), Macrodata Labs' public benchmark of **100** robot and egocentric episodes with **743** human-annotated subtask segments. It scores **0.197** Segment F1, 0.610 matched-label accuracy, and **0.121** end-to-end semantic F1, roughly **1.8×** the 0.110 open baseline (Qwen 3.6 Flash). **Are those results good?** They are an encouraging first result: the end-to-end score is roughly **1.8× the open baseline**, and once a predicted segment matches a human segment, its label agrees **61% of the time**. The clearest opportunity is boundary detection—the 0.197 Segment F1 shows that predicted transitions do not yet consistently align with human segmentation. Because the stricter 0.121 end-to-end score combines both boundary and label quality, the pipeline is already useful for bootstrapping annotations at scale while leaving substantial room for improvement. Credit to Macrodata for the benchmark, annotation protocol, and segmentation recipes we incorporated. **Reproduce the run.** Benchmark dataset: [macrodata/WGO-Bench](https://huggingface.co/datasets/macrodata/WGO-Bench). Annotated output (full 100 episodes): [pepijn223/wgo_bench_lerobot_subtask_qwen_full](https://huggingface.co/datasets/pepijn223/wgo_bench_lerobot_subtask_qwen_full). ### Where the ideas come from The pipeline stands on prior work, roughly one influence per module: | Feature | Source | |:---|:---| | Subtask granularity, "how, not what", interjection taxonomy | [Hi Robot](https://arxiv.org/abs/2502.19417), [π0.7](https://pi.website/pi07) | | Timestamped contact sheets, atomic segmentation, WGO-Bench | [Macrodata Labs](https://macrodata.co/blog/annotating-robot-video-subtasks) | | Contact-sheet dense captioning | [Scale AI](https://labs.scale.com/blog/path-to-large-scale-dense-video-captioning) | | Memory compression (minimal info, keep outcomes, drop attributes) | [MEM](https://arxiv.org/abs/2603.03596) | | Grounded VQA (bounding boxes, keypoints) | [ECoT](https://arxiv.org/abs/2407.08693), [Steerable VLA Policies](https://arxiv.org/abs/2602.13193) | The pipeline is a first step for automatic language annotations. It is built to grow. A new module (trajectory traces, affordances), a new prompt template, or a quality fix to an existing module is a very welcome PR. Modules live under src/lerobot/annotations/steerable_pipeline/modules/. Once those rows exist, the next chapter shows how recipes select them in time and turn them into training samples. ## Recipes: from rows to training samples Across robot learning, task labels, step annotations, memory targets, and high-level reasoning are often represented differently for each dataset and policy. This format is our best effort to consolidate those patterns into one path from annotation to language training. We do not expect every decision here to be optimal for every future policy. We prioritize a stable, flexible boundary: the dataset records *what happened and when*, a recipe defines *what to learn from it*, and each policy decides *how to serialize and train on the resulting messages*. That separation keeps data annotation, System 1 training, and System 2 training from becoming one fixed pipeline. You can revise an annotation module without redesigning the policy, change a training objective without rewriting the dataset, or reuse the same rendered messages with a different model processor. Storing language is therefore only half the problem. At training time, we must decide *which* rows a sample sees and *how* they become a conversation. A memory-update objective needs the previous memory and completed subtask; VQA needs a question, image, and answer; low-level control needs the task and active subtask. Same dataset, different samples. A **recipe** expresses that choice as YAML config, and the renderer emits policy-agnostic messages rather than applying a tokenizer template. [Figure: Three layers: data → recipe → training format] Follow one value through all three layers: a row selected from parquet, a temporal binding inserted into a message template, and the rendered target a policy processor serializes. ### Anatomy of a recipe Recipes are built from a few small primitives: 1. An **annotation row** stores a value such as a subtask, memory, or VQA event. 2. A **resolver** selects a row relative to the sampled frame `t`. 3. A **binding** gives that selected value a local name. 4. A **message** inserts bound values into a user or assistant turn. 5. `target: true` marks a turn for supervised generation by the policy. 6. `stream` labels a turn as `high_level` or `low_level` metadata; the policy processor decides what that label means. The stream label does not select a separate model or loss by itself. PI052 interprets a target as language-generation supervision and any `low_level` turn as enabling action-flow supervision, so one sample can train language, actions, or both through the same shared VLM. Here is a joint text-and-action objective: ```yaml bindings: active_subtask: "active_at(t, style=subtask)" messages: - { role: user, content: "${task}", stream: high_level } - { role: assistant, content: "${active_subtask}", stream: low_level, target: true } ``` For one sampled frame, the renderer runs `active_at(...)`, stores the result as `active_subtask`, substitutes it into the assistant message, and records that message as a target. PI052 uses the target for language loss and the `low_level` label for action-flow loss. A recipe containing only targeted `high_level` turns trains language generation without action flow. `${task}` is a built-in binding: it uses a deterministic `task_aug` rephrasing when available and otherwise falls back to the canonical task string. ### Bindings and temporal resolvers Resolvers describe *which value to select*, not how that value is tokenized: - `active_at(t, style=subtask)`: the persistent `subtask` row in effect at time `t`. - `nth_prev(style=memory, offset=1)`: the previous emitted memory, not the previous frame. - `nth_next(style=subtask, offset=1)`: the next subtask transition after the active one. - `emitted_at(t, style=interjection)`: an event row on exactly this frame. [Figure: What each binding resolves to] Scrub to a frame, predict each result, then reveal the answers. active_at reads current persistent state, nth_prev walks between earlier transitions, and emitted_at only matches an event on the exact frame. ### Blends: one dataset, many objectives A recipe can branch into a weighted **blend** of sub-recipes. On ordinary frames, the sample index deterministically selects one branch according to those weights, so the same index always gets the same objective. Frames carrying sparse VQA annotations are the exception: a matching `ask_vqa*` branch takes priority over the weighted draw so the label is not wasted. [Figure: Recipe to messages] ### View-dependent resolution for VQA With multiple cameras, a `(vqa, user)`/`(vqa, assistant)` pair can fire per camera at the same timestamp. A `camera=` filter (parallel to `role=`) disambiguates, with one sub-recipe and image block per camera: ```yaml ask_vqa_top: bindings: vqa_query: "emitted_at(t, style=vqa, role=user, camera=observation.images.top)" vqa: "emitted_at(t, style=vqa, role=assistant, camera=observation.images.top)" messages: - role: user stream: high_level if_present: vqa_query content: - { type: image, feature: observation.images.top } - { type: text, text: "${vqa_query}" } - { role: assistant, content: "${vqa}", stream: high_level, target: true, if_present: vqa } ``` ### Tools: when the robot calls a function Sometimes the right action is not a motor command but to *say* something, capture an extra observation, or log an event. LeRobot models this the way LLMs do: with **tool calls**. An assistant turn emits a structured invocation like `say(text="On it.")`; the dataset stores it and the policy is trained to generate it. Actually dispatching a generated call to a real implementation at inference is a separate layer we return to below. Tools split into two layers: the **catalog** (what *can* be called) and the **invocations** (an actual call, with arguments, on a specific frame). #### The catalog The catalog is a list of OpenAI-style function schemas, stored at `meta/info.json["tools"]`: A function schema is a JSON description of a callable (name, parameters, types). A policy processor decides how messages and tools become tokens. PI052, whose PaliGemma tokenizer has no chat template, flattens say calls to <say>...</say> markers. ```json { "tools": [ { "type": "function", "function": { "name": "say", "description": "Speak a short utterance to the user via the TTS executor.", "parameters": { "type": "object", "properties": { "text": { "type": "string", "description": "The verbatim text to speak." } }, "required": ["text"] } } } ] } ``` You read it through the metadata accessor, which falls back to `DEFAULT_TOOLS` (currently the single canonical `say` schema) when a dataset declares none, so chat-template consumers keep working with no configuration: ```python from lerobot.datasets.dataset_metadata import LeRobotDatasetMetadata meta = LeRobotDatasetMetadata(repo_id="pepijn/super_poulain_final_annotations") tools = meta.tools # list[dict] of OpenAI tool schemas, DEFAULT_TOOLS if none declared prompt_str = tokenizer.apply_chat_template(sample["messages"], tools=meta.tools, tokenize=False) print(prompt_str) ``` The final line shows the exact serialized prompt the tokenizer receives. #### The invocation, stored per row The actual call is stored per frame, on assistant atoms in `language_events`: ```python { "role": "assistant", "content": null, "style": null, "camera": null, "tool_calls": [ { "type": "function", "function": { "name": "say", "arguments": { "text": "On it." } } } ] } ``` Recipes splice these into a rendered turn with `tool_calls_from`. The shipped speech recipe creates a tool-call-only assistant target: ```yaml user_interjection_response: bindings: interjection: "emitted_at(t, style=interjection)" speech: "emitted_at(t, role=assistant, tool_name=say)" messages: - { role: user, content: "${task}", stream: high_level } - { role: user, content: "${interjection}", stream: high_level, if_present: interjection } - { role: assistant, stream: high_level, target: true, if_present: speech, tool_calls_from: speech } ``` The renderer attaches the structured call to the message. PI052's processor serializes it as a `...` text target, and its runtime can parse that marker back out. Runtime execution is not shipped yet: LeRobot does not yet include a tool registry or TTS dispatcher, so parsed `say` text is not spoken. For now, declare a schema to train generation; implementing and dispatching the call is an open extension described in the [tool-calling docs](https://huggingface.co/docs/lerobot/main/en/tools). ## Part 2 preview: policies in action This post stops at the policy-agnostic training example. **Part 2: Language-Conditioned Robot Policies**, coming later, follows that example into PI052: how System 2 generates language, how System 1 turns a subtask into actions, how the runtime connects them, and what happens in RoboCasa. Here is a preview of the simulation experiments: [Video demonstration]Microwave [Video demonstration]Cabinet [Video demonstration]Make coffee ## Get involved This release is a foundation, not a finish line. The dataset format, annotation pipeline, and recipes were built to be extended, and the most useful next language style or training objective may come from your data. ### Three ways to contribute **1. Annotate a dataset.** Grab an existing LeRobot dataset, run `lerobot-annotate` through [HF Jobs](https://huggingface.co/docs/lerobot/main/en/annotation_pipeline), and inspect the generated subtasks, plans, memory, and VQA in `.annotate_staging/` and the two language columns. Share the annotated dataset and what the pipeline got wrong. **2. Write a recipe.** Combine the same annotations into a new objective, inspect the rendered messages, and share the YAML. Recipes deliberately stay independent of tokenizers and model implementations. **3. Add a pipeline module or tool.** Add trajectory traces, affordances, grasp points, a sharper grounding flow, or a prompt-quality fix to `plan` / `interjections` / `vqa`. The tool-execution layer that turns a generated `say` into real speech is also still open. ### Why this matters Robotics is deeply multimodal, but model progress depends on data that makes the right intermediate concepts visible. Putting language beside robot trajectories gives researchers a common substrate for testing how subtasks, memory, grounded answers, and interaction affect learning. So annotate something, train something, break something, and tell us all about it. Share what you build on the [LeRobot Discord](https://discord.com/invite/q8Dzzpym3f) or tag us on the Hub. Nothing makes our day like seeing the datasets and models you create. #### Code and docs - **Language columns and recipes** [docs](https://huggingface.co/docs/lerobot/main/en/language_and_recipes) - **Annotation pipeline** [docs](https://huggingface.co/docs/lerobot/main/en/annotation_pipeline) - **Tools** [docs](https://huggingface.co/docs/lerobot/main/en/tools) #### Papers - **π0.5** Black et al. (2025). *π0.5: A Vision-Language-Action Model with Open-World Generalization.* [arxiv.org/abs/2504.16054](https://arxiv.org/abs/2504.16054) - **Hi Robot** Shi et al. (2025). *Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models.* [arxiv.org/abs/2502.19417](https://arxiv.org/abs/2502.19417) - **Pi0.7** Physical Intelligence (2025). *π0.7.* [pi.website/pi07](https://pi.website/pi07) - **MEM** Torne et al. (2026). *Memory for Vision-Language-Action Models.* [arxiv.org/abs/2603.03596](https://arxiv.org/abs/2603.03596) - **ECoT** Zawalski et al. (2024). *Robotic Control via Embodied Chain-of-Thought Reasoning.* [arxiv.org/abs/2407.08693](https://arxiv.org/abs/2407.08693) - **Steerable VLA Policies** Chen et al. (2026). *Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Control.* [arxiv.org/abs/2602.13193](https://arxiv.org/abs/2602.13193)