Ensemble Machine Learning Approach to Multimodal Deception Detection

Abstract

Demand for reliable deception detection systems has stimulated the application of machine learning techniques to behavioural analysis. Conventional approaches either rely on single or multiple modalities, including language, facial expression, or gestures. The single modality may not adequately capture the complex nature of deceptive behaviour, while the multimodal approaches also suffer from inaccuracies due the inability of a single algorithms to be able to effectively capture different aspects of the human behavioural indicators. To address thislimitation, an ensemble machine learning framework that integrates linguistic and facial cues for multimodal deception detection was developed. The framework combines psycholinguistic features extracted from real life dataset obtained from YouTube, using facial cues derived from the Facial Action Coding System (FACS) and linguistic cues gotten from the Linguistic Inquiry and Word Count (LIWC). An ensemble machine learning approach was employed, using the Random Forest classifier to learn facial deceptive traits, Support Vector Machine (SVM) to learn deceptive linguistic patterns, and fusing the predicted outputs by using the Extreme Gradient Boosting (XGBoost) asthe meta-classifier for producing the final classification decision. Experimental evaluation showed that the ensemble multimodal framework outperformed the individual classifiers across multiple performance metrics. The proposed system achieved an overall classification accuracy of 89.8%, an F1- score of 89.8%, and an Area Under the Receiver Operating Characteristic Curve (AUC) of 0.93, indicating strong discriminative capability between deceptive and truthful instances. This study affirms the effectiveness of ensemble strategies for improving deception detection accuracy by using modality-specific classifiers in multimodal feature analysis