Library robot: picking and stacking books.
After the hackathon we bought the robot and started building library applications on it. The headline result is a good one: a plain ACT policy generalized further than I expected, to books it had never seen and to books it could only partly see.
Why books
Books are a sensible step up from bread. They are rigid, which removes the deformable-grasp problem entirely, but they introduce variety in its place: books differ enormously in size, thickness, cover design and how they sit on a surface. That makes them a good instrument for asking whether a policy has learned the task or just memorized the demonstration.
How far it generalized
Same approach as the pen task: hold the policy fixed and change one thing at a time. Both of these are tests that usually break a policy which has simply latched onto a visual template.
| Factor | Condition | ACT |
|---|---|---|
| Baseline | Trained book, fully visible | Works |
| Book identity | Unseen book, different cover and proportions | Works |
| Visibility | Book partially occluded | Works |
The occlusion row is the more interesting of the two. A policy keyed to a memorized appearance tends to fail as soon as that appearance is incomplete, so surviving partial occlusion suggests it picked up something closer to the shape and the graspable edge than to a specific cover.
Worth holding this next to the desk cleanup experiment, where the same architecture on a smaller object failed to generalize across something as simple as color. ACT is not uniformly brittle or uniformly robust; what it latches onto depends on the task and the data, which is exactly why probing it matters.