Predicting Stock Market Volatility: a Comparative Analysis of Machine Learning Models and GARCH

Tan, Choon Yen (2025) Predicting Stock Market Volatility: a Comparative Analysis of Machine Learning Models and GARCH. Masters thesis, Tunku Abdul Rahman University of Management and Technology.

Text 10 Tan Choon Yen (MIM).pdf Restricted to Registered users only Download (2MB)

Abstract

Accurate volatility forecasting is crucial for risk management, asset pricing, and portfolio optimization. However, volatility prediction models may exhibit considerable performance variations across different market environments due to differences in liquidity, regulation, information flow, and investor behaviour. This study addresses the gap in existing literature by comparing the predictive performance of traditional econometric models (GARCH and GARCH-X) and advanced machine learning models (Random Forest, Support Vector Machine, and Gradient Boosting) across developed (United States, United Kingdom, Hong Kong, Japan) and emerging (Malaysia, Indonesia) stock markets. Utilizing daily stock market data from six major indices—S&P 500, FTSE 100, Hang Seng Index, Nikkei 225, FBM KLCI, and Jakarta Composite Index—this research examines the relative accuracy of volatility predictions. The performance of each model is assessed using multiple error metrics, including Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Quasi-Likelihood Loss (QLIKE). Additionally, feature importance analysis identifies critical variables influencing volatility predictions, such as historical volatility, trading volume, interest rates, and crisis events. Empirical results reveal that machine learning models, particularly Gradient Boosting and Random Forest, generally outperform traditional GARCH models, demonstrating superior ability to handle nonlinear patterns and incorporate diverse market information. The predictive advantage of machine learning methods is especially pronounced in emerging markets characterized by structural instabilities and heightened economic uncertainty. These findings highlight significant deviations from the semi-strong form of market efficiency, suggesting market inefficiencies due to delayed information integration. By demonstrating the practical benefits of incorporating advanced modelling techniques, this study contributes valuable insights for financial practitioners, policymakers, and researchers. The research also supports broader economic stability and growth aligned with Sustainable Development Goals, highlighting the importance of accurate risk modelling in fostering sustainable economic environments