Double Pendulum Control

An unactuated pendulum on a sliding cart is a classic controls problem, and this two-link version extends it. You can try to swing both links up by pushing the cart yourself, but the dynamics are too fast for human reaction time. A TQC reinforcement-learning policy does it with a high rate of success. Sometimes learned control policies are the only way to achieve stability! Built the rig? Flash a controller onto an ESP32-S3.

Controller

Manual: drag the cart. AI: the network drives to the target; picking a target hands it control.

Disturbances

Delay holds back the force; noise corrupts what the network measures, never the force. It was trained with neither, so 30 ms of delay or noise above 0.05 usually breaks it.

- Time to target (s)
Best Manual- AI-

Inside the network

Swipe sideways to see the whole network.

Each square is one neuron, violet when positive and green when negative. The ten inputs feed a 256-unit embedding, then two residual blocks (each widens to 1024 ReLU units and adds its result back through the skip), and a single number comes out. tanh squashes it to a force of at most 20 N.

About this demo

The policy was trained in n-cartpole with TQC (Truncated Quantile Critics), an off-policy actor-critic that learns a whole distribution of returns rather than a single value. At every 10 ms step it sees the cart position and velocity, cos θ, sin θ and θ̇ for each link, and the target as cos of each link's goal angle (+1 up, −1 down). It outputs one horizontal force. Its reward peaks only when both links sit at their targets, still, with the cart near the center. During training the target changed every few seconds, and episodes started at every pose as well as at fully random states. That's why it can move between any two targets, and why it can catch a swing you started by hand.

"Reached" means both links within 0.3 rad of their targets and slower than 1.5 rad/s for half a second. That's the same test the training repo's evaluation uses. The clock restarts at every reset and every new target. A time counts as AI only if you never pushed the cart yourself, and as manual only if the network never drove. Hitting either end of the rail ends the run. The network trained on a ±0.5 m rail; this one runs to ±1 m so there's room to swing by hand, but past half a meter from center the network is outside anything it saw in training.

Have the hardware? Install the firmware on an ESP32-S3 from your browser and load the model onto a real cart.

The simulation uses the exact plant the network was trained on, except the rail.