Mal-AAEGAN: Adversarial Autoencoder with Double-Critic WGAN-GP for Malware Generation

Abstract

Machine-learning (ML) malware detectors are known to be vulnerable to adversarial samples. Generative adversarial networks (GANs) such as MalGAN, Mal-LSGAN, and our previous Mal-D2GAN can craft adversarial malware to probe and harden these detectors, yet they operate directly on a high-dimensional, discrete binary feature space, which leads to unstable training, vanishing gradients, and adversarial samples of limited diversity. To address these problems we propose Mal-AAEGAN. First, an adversarial autoencoder (AAE) learns a low-dimensional, regularized and continuous latent representation of the $M$-dimensional binary API feature vector, and the generator performs an additive perturbation in this latent space rather than in the raw binary space. Second, training follows the Wasserstein GAN with gradient penalty (WGAN-GP) scheme using two critics: a mimic-black-box critic that inherits the role of the substitute detector and a naive critic that inherits the role of the additional detector. A decode-binarize-OR pipeline guarantees that the original API calls of the malware are preserved. The model is evaluated on two malware datasets of different input dimensionality ($M\in\{139,215\}$) against eight common ML detectors and compared, under an identical protocol, with MalGAN, Mal-LSGAN and Mal-D2GAN. Mal-AAEGAN trains stably in the continuous latent space and attains an evasion rate on par with or higher than these binary-space baselines across both datasets, while its generator stays responsive to the noise input rather than reducing to a single fixed perturbation per malware.