Online Control-Informed Learning
1Purdue University · Transactions on Machine Learning Research, 2025
A different part of the system is left unknown each time. The pendulum and rocket must recover the objective function from one demonstration; the cart-pole is given neither dynamics nor objective; the quadrotor must recover its dynamics, and gets three randomised input sequences to do it.
Abstract
We propose Online Control-Informed Learning, a framework that employs well-established optimal control and state estimation techniques to solve a broad class of learning tasks online. We consider any robot as a tunable optimal control system, parameterised by tunable parameters within its dynamics, its policy and its objective function.
Two parts do the work. A gradient generator gives the exact derivative of the trajectory with respect to those parameters, by differentiating through Pontryagin's Maximum Principle. An online parameter estimator based on the extended Kalman filter then corrects them as each measurement arrives.
This effectively handles practical issues in machine learning such as noisy measurement data, online learning, and data efficiency, across three modes: online imitation learning, online system identification, and policy tuning on the fly. Everything below is generated from our own code and data.
Method
One Observation, One Correction
Each measurement passes through the same five steps. Tap or hover any block to read what it does.
Interactive
Wrong Start, Right Finish
Drag the slider to move through training. The dashed shape is the demonstration; the solid one is the trajectory ξ(θ̂) the current estimate produces. At the far left the estimate has seen no data, so the two look nothing alike.
Results
Less Data, Lower Loss
OCIL drives the loss down from the first few measurements. The other methods need hundreds of times more data to reach the same level, and some never get there at all.
For each system the target is a loss the leading methods all reach. The bars show how much data each one needs to get there. A faded bar with a plus sign never got there at all, and runs to the end of its own budget. The axis is logarithmic.
Online computational performance
One Update, One Period
Learning online means absorbing each measurement before the next one arrives. We report the time one iteration takes as a share of the sampling period. Every case leaves headroom, so OCIL has the capability to run in an online fashion.
Tables 2 and 3. Bars are the mean iteration time as a percentage of that system's sampling period.
Citation
@article{liang2025online,
title = {Online Control-Informed Learning},
author = {Liang, Zihao and Zhou, Tianyu and Lu, Zehui and Mou, Shaoshuai},
journal = {Transactions on Machine Learning Research},
issn = {2835-8856},
year = {2025},
url = {https://openreview.net/forum?id=LDzvZEVl5H}
}