this adds a tool to “grade” robot demo episodes by analyzing blur, collisions, and movement smoothness, then filters the bad ones out of the dataset. Seems like a pragmatic way to tackle the data quality problem in robotics would be great to see how much it moves the needle in real training runs.
machinelearning•2mo ago
Training is expensive so I wouldn't necessarily call it "pragmatic".
- the tool's goal is actually to provide a lightweight, practical way to avoid wasting training cycles on bad data.
Evals for robotics are also expensive.
- validation loss is a poor proxy of robot performance because success is underconstrained by imitation learning data
- most robot evals today are either done in sim (which at best serves as a proxy) or by a human scoring success in the real world (which is expensive).
It's great if you have evals and want to backtrack (we're building tools for that too) but you definitely don't want to discover you have bad data after all that effort (learned that the hard way, multiple times).
The metrics the tool scores vary from tedious to impossible for a human to sanity check so there's some non-obvious practical value in automating some of it.
marshavoidance•3mo ago
machinelearning•2mo ago
- the tool's goal is actually to provide a lightweight, practical way to avoid wasting training cycles on bad data.
Evals for robotics are also expensive.
- validation loss is a poor proxy of robot performance because success is underconstrained by imitation learning data
- most robot evals today are either done in sim (which at best serves as a proxy) or by a human scoring success in the real world (which is expensive).
It's great if you have evals and want to backtrack (we're building tools for that too) but you definitely don't want to discover you have bad data after all that effort (learned that the hard way, multiple times).
The metrics the tool scores vary from tedious to impossible for a human to sanity check so there's some non-obvious practical value in automating some of it.