5 DOF chess robot

When my friend Marco and I flew to NYC for the largest robot arm hackathon in the world hosted by Betaworks, we didn't expect to outright win it, much less teach a robot to play chess after we'd ruled it out as a project due to the uncertainty around generalizing piece movement.

However, after forming a team with Carra Wu, Soham Bafana, and later being joined by Talha Khan, the prevailing sentiment leaned towards going big (or going home).

Raju, as we named the project, was an SO-100 arm with 5 DOFs and a Stockfish-powered backend. Estimating the board state was handled by a custom YOLO model trained purely on our board from a set camera view, and the arm itself operated on a custom ACT model. Back at Duke, I'd pitched the idea of teaching the arm to move a chosen game piece by training it on action/video data where the target was highlighted with a red circle (and blue circle for where to place). Neither of us had ever seen this done before, so we had ruled it out as the lack of precedent made it uncertain. But with our team onboard, we gave it a try.

We overfit our training data to the earliest opening moves to achieve the best odds of Raju being able to move during our demo (tele-operating a dataset of pawn, bishop, and knight moves that Stockfish would often pick). Video frames were intercepted by OpenCV and overlaid with the targeting circles. To help with dexterity, I redesigned and printed a new manipulator with cupped grippers and rubber covers to center pieces and avoid slipping.

By morning, Raju was mostly working, and we were able to implement basic piece taking (in theory) by using the generalized red-to-blue movement targets to move the taken piece off the board. To qualify for another potential prize, I wrote code to trash talk with Gemini and Fish Audio based on the Stockfish game analysis.

Raju took first place in the hackathon, and we went home with the grand prize. All in all, a great experience and I returned home feeling more optimistic about the future of tele-op training for robots. To read the devpost I scraped together in the final hour, read it here.