Computer Vision Based Sign Language Recognition System

Lim, Ze Jun (2020) Computer Vision Based Sign Language Recognition System. Final Year Project (Bachelor), Tunku Abdul Rahman University College.

Text LimZeJun_Full Text.pdf Restricted to Registered users only Download (7MB)

Abstract

Sign language is a language that uses body movements, gestures, and facial expressions to convey messages or information. Sign languages are commonly used among hearing-impaired as well as mutism community. However, static gesture recognition system is not capable of recognizing temporal content of gesture that contains movement. The main objective of this project is to develop a vision based American Sign Language (ASL) recognition system that has the capability to recognize the ASL that contains temporal feature. American Sign Language Lexicon Video Dataset (ASLLVD) is used in this project. Out of 3300 signs in the dataset, only 5 gestures in used during the neural network and Support Vector Machine (SVM) video sequence training process. Various image processing techniques such as GrabCut background subtraction, grayscaling, and image resizing operation are performed on the video dataset. In order to mitigate the temporal and spatial complexity, mean structural similarity index (MMSI) technique is used to reduce the temporal length of the video dataset while reserving the video portion that has the most information. Feature extraction of this dataset is done using multiple Convolutional Neural Network (CNN) models for instance InceptionResnetV2, VGG19, DenseNet201 and MobiliNetV2. Among these CNN models, the best validation accuracy of 15.24% is obtained by InceptionResnetV2 model. In summary, the model is too overfit due to small dataset size. To solve the overfitting issue, SVM is also used alongside Histogram of Oriented Gradient (HOG) descriptor. The best accuracy obtained from SVM method is 39.12%.