SpotNLP | Adam Imdieke

GitHub Repository

Overview

This project involves developing a natural language interface for the Boston Dynamics Spot robot, enabling users to command the robot using everyday language. By integrating large language models (LLMs) and gesture detection with Spot’s control systems, we aim to facilitate intuitive interactions and enhance the robot’s usability in various environments.

Human Following / Gesture Recognition

Recorded SpotNLP demonstration: Spot follows the controller, saves semantic keypoints, and navigates using contextual information about the environment.

Spot follows a selected person using segmentation models. XMem propagates the initial mask across frames to help maintain the track through distractions and occlusions in the recorded demonstrations. Spot also recognizes gestures to start and stop following.

MediaPipe initializes a segmentation mask when a person makes a T pose. XMem propagates the mask over time to help maintain the track when other people enter the scene or the controller is temporarily occluded.

SpotNLP can recognize gestures to start and stop following. The controller can make a T pose to start following or use a stop gesture to pause. Three further gestures adjust the following distance: close, medium, and far. MediaPipe pose estimates are combined into a history vector and passed to an MLP gesture classifier.

Natural Language Interface

SpotNLP enables natural-language commands for longer task sequences. In this demo, the user asks Spot to follow them, save semantic keypoints with contextual information, and navigate back to those keypoints. The user can also ask about the environment and refer to previous locations using the semantic map. Future work includes pointing gestures for more precise references to keypoints and manipulation with Spot's arm.

The videos above are recorded demonstrations of the SpotNLP system. They show specific behaviors in their recorded settings, not a general reliability or safety evaluation.