Computer Vision Based Sign Language Recognition System



Lim, Ze Jun (2020) Computer Vision Based Sign Language Recognition System. Final Year Project (Bachelor), Tunku Abdul Rahman University College.

[img] Text
LimZeJun_Full Text.pdf
Restricted to Registered users only

Download (7MB)


Sign language is a language that uses body movements, gestures, and facial expressions to convey messages or information. Sign languages are commonly used among hearing-impaired as well as mutism community. However, static gesture recognition system is not capable of recognizing temporal content of gesture that contains movement. The main objective of this project is to develop a vision based American Sign Language (ASL) recognition system that has the capability to recognize the ASL that contains temporal feature. American Sign Language Lexicon Video Dataset (ASLLVD) is used in this project. Out of 3300 signs in the dataset, only 5 gestures in used during the neural network and Support Vector Machine (SVM) video sequence training process. Various image processing techniques such as GrabCut background subtraction, grayscaling, and image resizing operation are performed on the video dataset. In order to mitigate the temporal and spatial complexity, mean structural similarity index (MMSI) technique is used to reduce the temporal length of the video dataset while reserving the video portion that has the most information. Feature extraction of this dataset is done using multiple Convolutional Neural Network (CNN) models for instance InceptionResnetV2, VGG19, DenseNet201 and MobiliNetV2. Among these CNN models, the best validation accuracy of 15.24% is obtained by InceptionResnetV2 model. In summary, the model is too overfit due to small dataset size. To solve the overfitting issue, SVM is also used alongside Histogram of Oriented Gradient (HOG) descriptor. The best accuracy obtained from SVM method is 39.12%.

Item Type: Final Year Project
Subjects: Technology > Mechanical engineering and machinery
Technology > Electrical engineering. Electronics engineering
Faculties: Faculty of Engineering and Technology > Bachelor of Mechatronics Engineering with Honours
Depositing User: Library Staff
Date Deposited: 24 Apr 2020 16:17
Last Modified: 19 Oct 2020 09:14