Gesture Recognition-Malaysian Sign Language Recognition with Convolutional Neural Network

Liew, Alison Shu Lien (2020) Gesture Recognition-Malaysian Sign Language Recognition with Convolutional Neural Network. Final Year Project (Bachelor), Tunku Abdul Rahman University College.

Text Alison Liew Shu Lien_Fulltext.pdf Restricted to Registered users only Download (2MB)

Abstract

Sign language is a communication medium for the deaf and vocally impaired. However, this language is not widely known or practised in public due to the deaf community being a minority and it takes time to learn and skilled manpower to assist the deaf in public interaction. Thus, project aims to produce a Malaysian Sign Language Recognition (MSLR) application to recognise MSL alphabets to help normal people communicate with the deaf. The scope of this project covers 24 alphabets (excluding j and z) and dynamic background with variations in illumination, objects in the background and non-skin tone objects. The proposed methodology involves background subtraction to detect the moving hand, skin segmentation to segment hands based on skin tones using YCbCr colour space for robustness in illumination variation and a 2D Convolutional Neural Network (2D CNN) model for feature extraction and classification of 24 alphabets (excluding j and z). As MSL alphabets are derived from American Sign Language (ASL), the ASL FingerSpelling Dataset from the University of Surrey’s Center for Vision, Speech and Signal Processing is used for model training and testing. Evaluation criteria include the model’s micro averages in Precision, Recall and F1-Score which could be found in the classification report. The test accuracy achieved is 79.54% with misclassifications on letters such as ‘E’ and ‘Q’ due to the signing orientation and similarity in gesture or finger articulation.