--- title: "Robot Imitation Learning: BC, DAgger, ACT, VLA, and RTC" subtitle: "From Supervised Policies to VLAs and Real-Time Action Execution" date: "2026-10-20" lang: en-US --- # Robot Imitation Learning: BC, DAgger, ACT, VLA, and RTC [Course home](https://github.com/zengxiong111/Foundation_of_Embodied_AI) · [简体中文](Data_Imitation_Learning_ZH.md) · [PDF](https://zengxiong111.github.io/zengxiong.github.io/files/Data_Imitation_Learning_EN.pdf) | Item | Details | |---|---| | Topic | BC, DAgger, ACT, VLA and RTC | | Revision | 2026-10-21 | | License | CC BY 4.0 · Xiong Zeng and contributors | **Scope.** These notes connect imitation-learning objectives, generative action-chunk policies, VLA action supervision, and continuous real-time execution under inference latency. Demonstration hardware, camera calibration, human-video retargeting, comprehensive deployment engineering, and hands-on experiments are not standalone topics here. Equations explain mechanisms and assumptions; not every equation reproduces a particular implementation line for line. **Reading path.** Behavior cloning and covariate shift → DAgger and corrective supervision → ACT, diffusion, and flow matching → VLA architectures and supervised action learning → temporal ensembling → inference-time and training-time real-time chunking. **Audience.** Researchers and graduate students working on robot manipulation, visuomotor policies, and the action heads of vision-language-action (VLA) models. ## Contents 1. **[Foundations: BC and DAgger.](#imitation-section-01)** Policy interface, supervised objectives, multimodality, distribution shift, interactive labeling, and recovery. 2. **[Action Chunks and Generative Imitation.](#imitation-section-02)** Chunk horizons, ACT, Diffusion Policy, Flow Matching, and algorithm selection. 3. **[From Imitation Learning to VLA.](#imitation-section-03)** Multimodal architectures, action outputs, supervision, and representative models. 4. **[Real-Time Chunk Execution and RTC.](#imitation-section-04)** Physical versus inference time, overlapping plans, inference-time guidance, and training-time conditioning. 5. [References](#imitation-section-05) ## 1. Foundations of Imitation Learning: BC and DAgger ### 1.1 What the policy learns: from observation histories to executable actions Imitation learning (IL) learns a closed-loop policy from expert demonstrations without requiring an explicitly designed reward or a known robot dynamics model. Let $s_t$ denote the latent physical state; $o_t$, camera and other exteroceptive observations; $q_t$, proprioception such as joint and hand states; and $g$, a task condition. A deployed policy often takes a history rather than a single image: $$ h_t=(o_{\le t},q_{\le t},g),\qquad a_t\sim\pi_\theta(\cdot\mid h_t),\qquad s_{t+1}\sim P(\cdot\mid s_t,u_t). $$ The policy action $a_t$ and the command actually applied to the plant $u_t$ are not necessarily identical: filtering, saturation, controllers, communication, and safety checks can intervene. One must define the action space precisely—for example, target joint positions $q^{\rm target}$, increments $\Delta q$, target end-effector pose $T^{\rm target}_{EE}$, or Cartesian increments $\Delta x_{EE}$. Absolute positions in training data cannot silently be interpreted as velocities or increments during deployment. Occlusion, unobserved contacts, and sensing latency make manipulation partially observable. Similar images may require different actions depending on contact history, the intended subtask, or joint feedback. Conditioning on history and proprioception can reduce ambiguity without guaranteeing recovery of the true hidden state. For a horizon $T$, define the state occupancy distribution induced by policy $\pi$: $$ d_t^\pi(s)=\Pr(s_t=s\mid\pi),\qquad d^\pi(s)=\frac{1}{T}\sum_{t=0}^{T-1}d_t^\pi(s). $$ The central distinction is simple: **BC fits expert actions on the expert occupancy $d^{\pi_E}$; DAgger collects supervision on states visited by the current learner, closer to $d^{\pi_\theta}$.** ### 1.2 Behavior Cloning: deterministic regression and probabilistic objectives Let $\mathcal D_E=\{(h_i,a_i^E)\}_{i=1}^N$ be expert examples. Deterministic BC commonly minimizes weighted squared error: $$ \theta^*=\arg\min_\theta\frac{1}{N}\sum_{i=1}^{N} \|f_\theta(h_i)-a_i^E\|_W^2,\qquad \|x\|_W^2=x^\top Wx,\quad W\succeq0. $$ This is appropriate when actions conditioned on $h$ are nearly unimodal and noise is modest. If $p_\theta(a\mid h)=\mathcal N(a;\mu_\theta(h),\Sigma)$ with fixed covariance $\Sigma$, maximum likelihood amounts, up to additive constants, to $$ \mathcal L_{\rm NLL}=\mathbb E_{\mathcal D_E}\!\left[ \tfrac12(a^E-\mu_\theta(h))^\top\Sigma^{-1}(a^E-\mu_\theta(h)) \right]+\mathrm{const}. $$ When covariance is learned, the Gaussian negative log-likelihood must also include $\tfrac12\log\det\Sigma_\theta(h)$; otherwise variance can grow without an appropriate penalty. An $L_1$ objective corresponds to a different observation-noise assumption, while categorical end-effector modes may use cross-entropy. A continuous gripper-position target should not be binarized without a reason. For action chunks ending near the end of an episode, padding is often required. Padding must be masked so it is not learned as an actual expert stop command: $$ \mathcal L_{\rm masked}= \frac{\sum_{i,k}m_{i,k}\,\ell(\hat a_{i,k},a^E_{i,k})} {\sum_{i,k}m_{i,k}+\varepsilon},\qquad m_{i,k}\in\{0,1\}. $$ Three prerequisites belong to the definition of the learning problem, not a separate data-engineering chapter: normalizers are estimated on the training split only; labels are causally aligned with observations available at decision time; and physical units, coordinate frames, and control modes match between training and execution. ### 1.3 Multimodal demonstrations: why a regression mean can be wrong Imagine two safe expert maneuvers around an obstacle, one on its left and one on its right. Under squared error, the Bayes-optimal point prediction is the conditional mean: $$ f^*(h)=\mathbb E[a^E\mid h]. $$ If feasible actions form separate modes $a^{(L)}$ and $a^{(R)}$, predicting $\big(a^{(L)}+a^{(R)}\big)/2$ may drive straight toward the obstacle. This is a **representation mismatch between multimodal actions and a single point estimate**, not merely insufficient network capacity. There are several solutions: give the policy intent information or longer history; introduce a latent variable to represent alternative coherent trajectories, as ACT does; or use a generative distribution model such as diffusion or flow matching. Better conditional density modeling does not, by itself, supply demonstrations of corrective actions in states the expert rarely visits. ### 1.4 Covariate shift and compounding errors BC is trained with states drawn from $d^{\pi_E}$ but deployed on $d^{\pi_\theta}$. A small reaching error may change fingertip contact, leading to slip, a different image, and additional errors. The policy can therefore fail closed-loop despite low held-out action regression error. A classic finite-horizon thought experiment assumes a probability $\epsilon$ of a *critical error* at an expert state. Once an early error sends the agent away from the expert trajectory, errors can compound over time. Under worst-case assumptions, accumulated cost can scale as $$ J(\pi_\theta)-J(\pi_E)=O(T^2\epsilon). $$ If supervision instead keeps the policy's single-step error small **on its own visitation distribution**, suitable bounded-loss and expert cost-to-go assumptions can yield a more favorable $O(T\epsilon)$ dependence. These are explanatory theoretical scalings, **not** formulas for robot task success rate: continuous-action MSE is not itself a probability of a critical error. ### 1.5 DAgger: obtain expert labels on learner-induced states DAgger, or Dataset Aggregation, specifies an *interactive supervision protocol* rather than a neural network architecture. At iteration $k$, a current policy visits states; an expert gives the preferred actions for those visited states; the labeled examples are aggregated; and the learner is trained again: $$ \begin{aligned} \pi_k^{\rm roll}&=\beta_k\pi_E+(1-\beta_k)\pi_k,\\ \mathcal D_{k+1}&=\mathcal D_k\cup \{(h,\pi_E(h)):h\sim d^{\pi_k^{\rm roll}}\},\\ \pi_{k+1}&\approx\arg\min_{\pi\in\Pi} \sum_{(h,a^E)\in\mathcal D_{k+1}}\ell(\pi(h),a^E). \end{aligned} $$ Here $\beta_k$ means the probability of selecting expert control **at a state/time step**. It does not mean that robot and expert command vectors should be interpolated; arbitrary interpolation can be unsafe during contact-mode transitions. Unlike simply collecting more successful demonstrations, DAgger's defining feature is that **the learner induces the states but the expert supplies their action labels**. The surrogate loss on the distribution visited in round $k$ is $$ L_k(\pi)=\mathbb E_{h\sim d^{\pi_k}}\,[\ell(\pi(h),\pi_E(h))]. $$ If the sequential supervised learner incurs regret $R_N$ over these changing loss functions, then $$ \frac1N\sum_{k=1}^N L_k(\pi_k) \leq \underbrace{\min_{\pi\in\Pi}\frac1N\sum_{k=1}^N L_k(\pi)}_{\epsilon_N} +\frac{R_N}{N}. $$ Connecting this self-visited-distribution risk to execution performance requires additional assumptions: reliable expert actions, bounded surrogate and task costs, an appropriate rollout mixture, and adequate coverage. No-regret learning alone does not establish a universal robot success-rate guarantee, nor does it remove the need for a safety supervisor. ### 1.6 Corrective demonstrations: learn recovery, not failed commands Failed robot episodes contain several qualitatively different labels: - **Valid actions before failure:** they may remain useful BC positives even if the episode ends unsuccessfully. - **Erroneous learner actions that caused failure:** keep them for diagnosis and state visitation, but do not blindly train them as expert targets. - **Expert corrections or recovery actions:** under a matching observation and action interface, these are informative supervision for recovery states. Practical expert interfaces include offline relabeling of learner-visited states, live human takeover, and querying an expert action while the current policy is running. An offline video may omit the contact force or hidden geometry needed for reliable relabeling. Dangerous or irrecoverable states may require reset or a safety stop rather than a forced action label. $$ \mathcal D_0\xrightarrow{\rm train}\pi_0 \xrightarrow{\rm rollout}\mathcal H_0 \xrightarrow{\rm expert\ label}\Delta\mathcal D_0 \xrightarrow{\rm aggregate}\mathcal D_1. $$ Records should distinguish $a_t^{\rm executed}$ from $a_t^{\rm expert}$, and note model version, intervention flag, and label confidence. A trajectory-level scalar reward or failure flag without statewise expert action labels is not standard DAgger; it belongs to a different interface, such as reinforcement learning or data filtering. ### 1.7 DAgger and generative policies are complementary | Method or protocol | Source of supervision | Main purpose | |:--|:--|:--| | Single-step BC | Fixed expert dataset | Fit expert actions by regression or likelihood | | ACT / chunked BC | Fixed expert dataset | Predict temporally coherent action chunks | | Diffusion / flow BC | Fixed expert dataset | Represent multimodal chunk distributions | | DAgger + supervised policy | Expert labels at learner-induced states | Add correction states and reduce distribution shift | Consequently, **DAgger + ACT** and **DAgger + Flow Policy** are meaningful combinations. ACT/flow specify how the policy represents an action sequence; DAgger specifies where additional labeled histories come from. For a chunk decoder, however, a single expert action does not supervise every time index of an $H$-step prediction. One needs chunk-level labels, a valid partial-label mask, or a revised objective. ## 2. Action Chunks and Generative Imitation Learning ### 2.1 Action chunking: prediction horizon versus execution horizon Given history $h_t$, a chunk policy predicts a sequence of future actions: $$ A_t=(a_{t\mid t},a_{t+1\mid t},\ldots,a_{t+H-1\mid t})\in\mathbb R^{H\times D_a}, \qquad A_t\sim p_\theta(\cdot\mid h_t). $$ The *prediction horizon* $H$ is not the number of actions that must be executed open-loop. An executor may consume only $K\leq H$ actions before querying a new observation and plan. Larger $K$ typically reduces inference frequency but slows reaction to disturbances; smaller $K$ improves feedback opportunities but adds computational and plan-stitching pressure. Chunking defines a **policy output interface**; ACT, diffusion, and flow matching define different representations and objectives for this interface. ### 2.2 ACT: a conditional VAE, Transformers, and temporal ensembling Action Chunking with Transformers (ACT) was introduced for fine-grained bimanual manipulation. Its conditional VAE has two distinct paths. **Training-side style encoder.** Given current robot state $q_t$ and a ground-truth future demonstration chunk $A_t^E$, an encoder infers a style latent variable: $$ q_\phi(z\mid q_t,A_t^E)=\mathcal N\!\left( \mu_\phi(q_t,A_t^E),\ \mathrm{diag}\,\sigma_\phi^2(q_t,A_t^E) \right),\qquad z=\mu_\phi+\sigma_\phi\odot\epsilon. $$ **Action decoder.** A visuomotor Transformer fuses multi-view visual features, robot proprioception, and $z$, and predicts an entire $H\times D_a$ action chunk in parallel: $$ \hat A_t=f_\theta(h_t,z),\qquad p(z)=\mathcal N(0,I). $$ In the original ACT architecture, a vision backbone converts each camera image into spatial tokens; the Transformer encoder combines these with robot and latent tokens. A decoder uses $H$ action queries to predict all future steps. ACT is not required to sample one action at a time autoregressively. The original training-side VAE encoder uses robot state and **expert action sequences**, rather than the camera images as its principal inputs. A representative loss for original ACT is masked $L_1$ reconstruction plus KL regularization: $$ \mathcal L_{\rm ACT}=\mathbb E\!\left[ \frac{\sum_{k=0}^{H-1}m_k\|\hat a_{t+k\mid t}-a^E_{t+k}\|_1} {\sum_km_k+\varepsilon} +\beta D_{\rm KL}\!\left(q_\phi(z\mid q_t,A_t^E)\|\mathcal N(0,I)\right) \right], $$ $$ D_{\rm KL}=\frac12\sum_j\left(\mu_j^2+\sigma_j^2-\log\sigma_j^2-1\right). $$ A small $\beta$ may create a mismatch between a highly informative posterior during training and prior-based inference; an excessively large $\beta$ can cause posterior collapse. **Training and inference are asymmetric:** the training encoder sees the expert future chunk, which is not available at inference. The original ACT deployment convention takes the prior mean $z=0$ for a deterministic rollout. Stochastic prior sampling is a possible extension, not the default rule used in the original paper. **Temporal ensembling** merges predictions created at different request times but targeting the *same physical execution instant* $t$. Let $\mathcal R(t)$ be those request times: $$ \bar a_t=\frac{\sum_{r\in\mathcal R(t)}w_{r,t}\hat a_{t\mid r}} {\sum_{r\in\mathcal R(t)}w_{r,t}},\qquad w_{r,t}=\exp[-m\,j(r,t)]. $$ If $j$ counts from the **oldest** prediction ($j=0$) to newer ones, this particular weighting favors older plans; reversing the ordering changes that interpretation. Temporal ensembling averages several proposed commands *for the same moment*; it is not an ordinary low-pass filter over consecutive physical times. Strong smoothing may obscure important contact onset, release, and slip events. ### 2.3 Diffusion Policy: learning a chunk distribution by denoising Let $A_0$ be a demonstration action chunk. A forward diffusion step $n$ produces a noisy version: $$ A_n=\sqrt{\bar\alpha_n}A_0+ \sqrt{1-\bar\alpha_n}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I). $$ A conditional noise predictor $\epsilon_\theta(A_n,n,h)$ is trained with the common simplified objective $$ \mathcal L_{\rm diff}= \mathbb E_{A_0,h,n,\epsilon} \|\epsilon-\epsilon_\theta(A_n,n,h)\|_2^2. $$ At inference, the policy starts from a noise chunk and iteratively denoises it into a sequence conditioned on the latest robot observations and task. Diffusion can represent alternative action trajectories rather than averaging them into one infeasible sequence. However, denoising needs model evaluations, and the sampling schedule, model size, and inference overhead determine latency. **A powerful generative model is not automatically a real-time executor.** ### 2.4 Flow Matching: a velocity field in action-generation space Flow matching constructs a path from a noise endpoint $A_0\sim p_0$ to a demonstration chunk $A_1\sim p_{\rm data}(\cdot\mid h)$. For a simple straight conditional path, $$ A_\tau=(1-\tau)A_0+\tau A_1,\qquad \tau\sim\mathcal U[0,1]. $$ The target conditional velocity along this path is $A_1-A_0$, yielding the conditional flow-matching objective $$ \mathcal L_{\rm FM}=\mathbb E_{h,A_0,A_1,\tau} \left\|v_\theta(A_\tau,\tau,h)-(A_1-A_0)\right\|_2^2. $$ Generation integrates the learned vector field: $$ \frac{\mathrm d A_\tau}{\mathrm d\tau}=v_\theta(A_\tau,\tau,h),\qquad A_{\tau=0}\sim p_0,\quad A_{\tau=1}\approx A^{\rm pred}. $$ The variable $\tau$ is *virtual generative time*, not the physical robot clock $t$. Correspondingly, $v_\theta$ is a velocity field for **moving action samples through probability space**, not a joint velocity $\dot q$ or end-effector linear velocity. Flow matching does not imply a one-step sampler: real-time behavior still depends on the neural model, numerical solver, and function evaluation count. ### 2.5 Selecting a model family | Policy family | Training target | Appropriate use | Principal caveat | |:--|:--|:--|:--| | Single-step BC | MSE/NLL of one $a_t$ | Nearly unimodal, high-frequency actions | Weak multimodal and longer-term modeling | | ACT | Chunk reconstruction + KL | Coherent fine manipulation from demos | Prior/posterior mismatch, chunk transitions | | Diffusion | Noise prediction on chunks | Multiple plausible trajectories | Iterative sampling delay | | Flow Matching | Generative vector field | Generative VLA action heads | ODE integration and inference latency | A model choice should start with four concrete questions: Are demonstrations multimodal? Is the intent identifiable from the supplied history? What physical command does the action tensor mean? What end-to-end decision latency can the robot tolerate? **Action-density modeling and state-distribution coverage are separate dimensions**: even a highly expressive generative policy may need DAgger, recovery demonstrations, or other interactive correction. ## 3. From Imitation Learning to Vision-Language-Action Models (VLA) ### 3.1 What is a VLA? Learning actions conditioned on vision and language A **vision-language-action model (VLA)** typically maps visual observations, language instructions, and robot proprioception to executable actions. In the notation established above, a general action-chunk interface is $$ A_t\sim p_\theta(\cdot\mid I_{\le t},q_{\le t},\ell,e),\qquad A_t=(a_{t\mid t},\ldots,a_{t+H-1\mid t}), $$ where $I_{\le t}$ denotes camera observations, $q_{\le t}$ the robot's joint, end-effector, or hand history, $\ell$ a natural-language task instruction, and $e$ the robot embodiment and action-space specification. Not every VLA explicitly uses every listed condition or predicts $H>1$ actions. This equation defines the **interface between task conditions and executable action**, not a mandatory implementation for every architecture. The key distinction is one of abstraction. **BC is a supervised learning objective; DAgger is an interactive data-aggregation protocol; ACT, diffusion, and flow matching are ways to represent action distributions; VLA denotes a class of policies that connect multimodal representations to robot actions.** A VLA action module can still be trained by exactly the imitation-learning losses discussed in Chapters 1–2. Simply adding language input to a small BC policy does not establish the cross-task generalization of a foundation model: broad data, pretrained representations, and architecture matter. ### 3.2 Architecture: connecting VLM representations to an action generator A common, but not universal, decomposition is $$ Z_t=f_{\phi}(I_{\le t},\ell),\qquad A_t\sim p_{\psi}(\cdot\mid Z_t,q_{\le t},e). $$ A vision-language encoder or backbone produces representations $Z_t$ of objects, scenes, and instructions. An action module conditions on those representations and proprioception to produce outputs in a specified robot action space. Modules may be jointly trained, frozen, or partially fine-tuned. Robot-specific decoding, command limits, and low-level controllers remain necessary: **understanding an instruction with a VLM is not equivalent to learning safe contacts or dexterous control**. Two common output interfaces reuse different ideas from Chapter 2: | Action interface | Supervision and output | Example | Main trade-off | |:--|:--|:--|:--| | Discrete action tokens | Quantize continuous controls and predict tokens autoregressively using cross-entropy | OpenVLA | Reuses language-model interfaces but introduces quantization and token-generation latency | | Continuous generative chunks | Generate $H$ real-valued actions conditioned on multimodal features, e.g. via flow matching | $\pi_0$, GR00T N1 | Natural for continuous high-dimensional control but adds generative compute and chunk-execution constraints | For a discretized expert-action sequence $z^E=(z_1^E,\ldots,z_M^E)$, an autoregressive objective is $$ \mathcal L_{\rm token}=-\mathbb E_{\mathcal D_E} \sum_{j=1}^{M}\log p_\theta(z_j^E\mid z_{