AI-Driven Hand Gesture Recognition for Robotic Arm Movements

Yep, Yong Kin (2025) AI-Driven Hand Gesture Recognition for Robotic Arm Movements. Final Year Project (Bachelor), Tunku Abdul Rahman University of Management and Technology.

Text Yep Yong Kin_FULL TEXT.pdf Restricted to Registered users only Download (16MB)

Abstract

This project presents the development of an AI-driven dynamic hand gesture recognition system to control both the speed and the directional movement of a KUKA robotic arm, with the goal of enhancing human-robot interaction (HRI) in industrial settings. Addresses the inefficiencies of traditional robotic control methods, which often lack intuitiveness and adaptability in dynamic environments. The proposed system recognizes dynamic hand gestures such as front, back, left, right, and neutral alongside speed commands, including fast, slow, and intermediate. The system utilizes MediaPipe for real-time hand landmark detection and OpenCV for video capture and processing. Gesture sequences are processed using a Bidirectional Long-Short-Term Memory (BiLSTM) network, which is trained on a custom dataset collected in varied environmental conditions to ensure robustness against lighting variations and background clutter. Software development was conducted in Python, while robot motion commands were executed using the KUKA Robot Language (KRL). The system successfully achieved real-time performance with signal latency ranging average 0.6s, meeting the objective of responsive and reliable control. The BiLSTM model demonstrated a high recognition accuracy of 98.57%, enabling intuitive and robust gesture-based control of the robotic arm. This approach improves operational safety, reduces physical strain on operators, and minimizes human error. Future work will aim to expand the gesture vocabulary, optimize system latency, and enhance integration into complex industrial environments.