Enhancing a channel estimation for fbmc/oqam using deep neural network / Nura Abdalrhman Alhajalsharief Mahmoud

Faculty of Electrical & Electronics Engineering Technology

Thesis (Doctor of Philosophy) -- Universiti Malaysia Pahang - 2026

Include bibliographical reference

Filter Bank Multicarrier with Offset Quadrature Amplitude Modulation (FBMC/OQAM) is a promising multicarrier modulation technique for sixth-generation (6G) networks that offers high spectral efficiency and cyclic prefix (CP) free transmission at reduced out-of-band emissions (OOBE) and resilience to asynchronous environments. However, this technique complicates the critical process of channel estimation (CE) in wireless communication due to the interference from a lack of orthogonality in the imaginary domain. Its performance in terms of system capacity, spectral efficiency, and data rates are enhanced through the integration of massive multiple-input multiple-output (m-MIMO), yet, at the reliance on accurate channel state information (CSI). While the least-squares (LS) estimator as a commonly employed interference approximation method (IAM) is computationally efficient, it suffers poor accuracy towards noise, multipath fading, and condition of low signal-to-noise ratio (SNR). With this in mind, the focus is shifted to deep neural networks (DNN) for an outstanding likelihood against the pinpointed shortcomings as a reliable and simplified solution to enhanced wireless CE performance. This is supported by the distributive technicalities of the graphics processing unit (GPU) for DNN applications in actual settings. The current study, therefore, introduces an improved IAM-LS (Interference approximation method) channel estimator based on the DNN approach (M-IAM-LS-DNN) for enhanced CE performance. The proposed model was trained and tested through MATLAB simulations under Rayleigh, and Rician fading channel and additive white Gaussian noise (AWGN) across various SNR levels and mobility scenarios including pedestrian (Ped A), road-to-vehicular (RTV), and vehicular (Veh A, Veh B) channels. Obtained results on the implementation of M-IAM-LS-DNN further confirm a 10% reduction in normalized mean square error (NMSE) and a 33% improvement in bit error rate (BER) against the conventional M-IAM-LS. The suitability of the proposed approach for applications is then substantiated by 8% and 12% lower computational complexities for the three and two hidden layers layouts, respectively. This has been achieved by eliminating the complex matrix computations with a concurrent improvement of BER, NMSE, and overall system capacity upon m-MIMO following integration with m-MIMO.