Non-Frontal-Face Deepfake Detection Using Neural Network

Wong, Jun Yen (2021) Non-Frontal-Face Deepfake Detection Using Neural Network. Final Year Project (Bachelor), Tunku Abdul Rahman University College.

Text Wong Jun Yen.pdf Restricted to Registered users only Download (2MB)

Abstract

Nowadays, Deepfake is easily generated by either computers and smartphones. As it grows more prominent day by day in the filming industry and social media, deepfake are becoming more and more realistic where it reaches a point the human eye is having difficulty identifying the highly advance deepfake manipulated video. Recently, deepfake are starting to tamper on videos that are non-frontal-face oriented. However, existing deepfake detectors do not perform well for non-frontal deepfake manipulated facial expression vides. This is due to most of the existing deepfake detectors are trained on the dataset of journalists talking directly to the camera. Clips with the target faces that are not facing directly to the camera, such as the side of the face facing the camera, have higher difficulty for neural network to detect than faces that are facing the camera. In this project, main objective is to create a deepfake detector that can detect deepfake manipulated facial expressions on non-frontal-face oriented videos using deep learning and identify the accuracy of the proposed model. MTCNN Face Detector and EfficientNet are selected as the backbone for non-frontal-face deepfake detection. Facial landmarks and SSIM difference masks are extracted from the Deepfake Detection Challenge dataset. The features are then pass to EfficientNet for training. After training, EfficientNet is used to predict the confidence score that the video has gone through deepfake manipulation, with 1 being FAKE, 0 being REAL. In the end, the highest mean confidence score generated through the frame-by by-frame method, 0.98978 for 1 video and achieve an accuracy of 0.842 with 2 maximum frame per video.