Leveraging Large Language Models for Interactive Text-to-ASL Translation: ASL Translation, Pose Extraction Module

Yap, Zi Yan (2025) Leveraging Large Language Models for Interactive Text-to-ASL Translation: ASL Translation, Pose Extraction Module. Final Year Project (Bachelor), Tunku Abdul Rahman University of Management and Technology.

Text RDS_Yap Zi Yan_Fulltext.pdf Restricted to Registered users only Download (3MB)

Abstract

This project addresses the communication barriers faced by the Deaf and Hard of Hearing (DHH) community by developing a system that translates text into American Sign Language (ASL) gloss and extracts pose landmarks for generating visual animations. The system comprises two core modules: the ASL Translation Module, which utilizes a large instruction-tuned language model to convert multilingual text into ASL gloss with strict adherence to linguistic rules, and the Pose Extraction Module, which employs MediaPipe to process merged videos and generate consistent and accurate pose data. Using Python, Hugging Face API, and MediaPipe, the system delivers reliable performance, as demonstrated through rigorous testing of both translation accuracy and pose extraction quality. Despite challenges such as limited vocabulary and hardware constraints for scaling, the project achieves its objectives of enhancing accessibility and inclusivity. It lays a strong foundation for future improvements, including vocabulary expansion, integration of non-manual markers, and scalability optimizations, ensuring its continued relevance in bridging communication gaps for the DHH community.