Memory-Resident Malware Detection via a Hybrid Deep Learning Framework
DOI:
https://doi.org/10.54654/isj.v3i26.1177Keywords:
Malware, hybrid framework, CIC-MalMem-2022, deeplearning, mô hình lan truyền mã độc SCIRSTóm tắt
Memory-resident malware detection is a critical cybersecurity challenge, particularly with stealth techniques like Living-off-the-Land (LotL). This paper proposes a hybrid deep learning framework to detect malware from memory-behavior data represented as fixed-length tabular features. The framework emphasizes an effective data processing pipeline rather than a complex model architecture. It has three stages: (1) feature selection and Z-score standardization using Extreme Gradient Boosting (XGBoost) and StandardScaler, (2) data balancing and cleaning using Synthetic Minority Over-sampling Technique (SMOTE) and Edited Nearest Neighbor (ENN), and (3) training a Transformer Encoder-based classifier to extract high-level non-linear representations from the stabilized feature space, utilizing robust Feed-Forward Networks, Layer Normalization, residual connections, and Focal Loss to enhance training stability under class imbalance. Training further employs a StepLR Scheduler and Early Stopping to ensure convergence and prevent overfitting. On the CIC-MalMem-2022 dataset, which comprises one benign class and 15 malware classes, the proposed framework achieves 76.62% Accuracy and 76.35% F1-score, outperforming traditional baselines. These results demonstrate the framework’s effectiveness for proactive malware defense based on memory behavioral analysis.
Downloads
References
A. Bensaoud, J. Kalita, and M. Bensaoud, “A survey of malware detection using deep learning,” Machine Learning with Applications, vol. 16, p. 100546, Jun. 2024. doi: doi.org/10.1016/j.mlwa.2024.100546.
T. Leng, Y. Pan, L. Zhao, A. Yu, Z. Zhu, L. Cai, and D. Meng, “MemInspect: Memory Forensics for Investigating Fileless Attacks,” in Proceedings of the IEEE 22nd International Conference on Trust, Security and Privacy in Computing and
Communications (TrustCom), Exeter, United Kingdom, 2023, NJ: IEEE, pp. 946–955. doi: doi.org/10.1109/TrustCom60117.2023.00134.
P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, “MeMalDet: A memory analysis-based malware detection framework using deep autoencoders and stacked ensemble under temporal evaluations,” Computers & Security, vol. 142, p. 103864, 2024. doi: doi.org/10.1016/j.cose.2024.103864.
T. Carrier, P. Victor, A. Tekeoglu, and A. H. Lashkari, “Detecting Obfuscated Malware using Memory Feature Engineering,” in Proceedings of the 8th International Conference on Information Systems Security and Privacy (ICISSP), Online Streaming, 2022, Portugal: SCITEPRESS, pp. 177–188. doi: doi.org/10.5220/0010908200003120.
S. R. Safavian and D. Landgrebe, “A Hybrid SMOTEENN Method for Malware Detection in Imbalanced Datasets,” Computers & Security, vol. 128, p. 102931, 2023. doi: doi.org/10.1016/j.cose.2023.102931.
A. Gaur, P. Mishra, P. Vinod, A. Singh, V. Varadharajan, U. Tupakula, and M. Conti, “vDefender: An explainable and introspectionbased approach for identifying emerging malware behaviour at hypervisor-layer in virtualization environment,” Computers and Electrical Engineering, vol. 120, Part B, p. 109742, 2024. doi: doi.org/10.1016/j.compeleceng.2024.109742.
V. Borisov, T. Leemann, K. Seßler, J. Haug, M. Pawelczyk, and G. Kasneci, “Deep Neural Networks and Tabular Data: A Survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 6, pp. 7499–7515, 2024. doi: doi.org/10.1109/TNNLS.2022.3229161.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need,” in Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA, 2017, NY: Curran Associates, Inc., pp. 5998–6008. doi: doi.org/10.48550/arXiv.1706.03762.
T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal Loss for Dense Object Detection,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017, NJ: IEEE, pp. 2999–3007. doi: doi.org/10.1109/ICCV.2017.324.
T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 2016, NY: ACM, pp. 785–794. doi: doi.org/10.1145/2939672.2939785.
K. S. Roy, T. Ahmed, P. B. Udas, M. E. Karim, and S. Majumdar, “MalHyStack: A hybrid stacked ensemble learning framework with feature engineering schemes for obfuscated malware analysis,” Intelligent Systems with Applications, vol. 20, p. 200283, 2023. doi: doi.org/10.1016/j.iswa.2023.200283.
J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016. doi: doi.org/10.48550/arXiv.1607.06450.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, NJ: IEEE, pp. 770–778. doi: doi.org/10.1109/CVPR.2016.90.
Y. Song, D. Zhang, J. Wang, Y. Wang, Y. Wang, and P. Ding, “Application of deep learning in malware detection: a review,” Journal of Big Data, vol. 12, no. 99, pp. 1–23, 2025. doi: doi.org/10.1186/s40537-025-00999-6.
A. Galli, V. La Gatta, V. Moscato, M. Postiglione, and G. Sperlì, “Explainability in AI-based behavioral malware detection systems,” Computers & Security, vol. 143, p. 103842, 2024. doi: doi.org/10.1016/j.cose.2024.103842.
M. M. Alani, A. Mashatan, and A. M. Miri, “XMal: A lightweight memory-based explainable obfuscated-malware detector,” Computers & Security, vol. 133, p. 103409, 2023. doi: doi.org/10.1016/j.cose.2023.103409.
T. A. Tuan, P. S. Nguyen, P. N. Van, N. D. Hai, P. D. Trung, N. T. K. Son, and H. V. Long, “A novel framework for cross platform malware detection via AFSP and ADASYN-based balancing,” Computers and Electrical Engineering, vol. 128, p. 110625, 2025. doi: doi.org/10.1016/j.compeleceng.2025.110625.
P. S. Nguyen, P. N. Van, H. V. Long, and P. D. Trung, “An Efficient Framework for MultiClass Malware Classification in Cloud Environments,” Journal of Science and Technology on Information Security, vol. 1, no. 24, pp. 1–10, 2023. doi: doi.org/10.54654/isj.v1i24.1092.
M. M. Abualhaj, S. Al-Khatib, N. Al Shafi, I. Qaddara, and A. Hyassat, “Utilizing gray wolf optimization algorithm in malware forensic investigation,” Journal of Computer and Cognitive Engineering, vol. 00, no. 00, pp. 1–12, 2025. doi: doi.org/10.47852/bonviewJCCE52025053.
S. S. Shafin, G. Karmakar, and I. Mareels, “Obfuscated memory malware detection in resource-constrained IoT devices for smart city applications,” Sensors, vol. 23, p. 5348, 2023. doi: doi.org/10.3390/s23115348.
Downloads
Published
How to Cite
Issue
Section
License
Proposed Policy for Journals That Offer Open Access
Authors who publish with this journal agree to the following terms:
1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
Proposed Policy for Journals That Offer Delayed Open Access
Authors who publish with this journal agree to the following terms:
1. Authors retain copyright and grant the journal right of first publication, with the work [SPECIFY PERIOD OF TIME] after publication simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
