arXiv · 2412.02540
Automatic State Machine Inference for Binary Protocol Reverse Engineering
Abstract
Protocol Reverse Engineering (PRE) is used to analyze protocols by inferring their structure and behavior. However, current PRE methods mainly focus on field identification within a single protocol and neglect Protocol State Machine (PSM) analysis in mixed protocol environments. This results in insufficient analysis of protocols' abnormal behavior and potential vulnerabilities, which are crucial for detecting and defending against new attack patterns. To address these challenges, we propose an automatic PSM inference framework for unknown protocols, including a fuzzy membership-based auto-converging DBSCAN algorithm for protocol format clustering, followed by a session clustering algorithm based on Needleman-Wunsch and K-Medoids algorithms to classify sessions by protocol type. Finally, we refine a probabilistic PSM algorithm to infer protocol states and the transition conditions between these states. Experimental results show that, compared with existing PRE techniques, our method can infer PSMs while enabling more precise classification of protocols.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Junhai Yang, Fenghua Li, Yixuan Zhang, Junhao Zhang, Liang Fang, Yunchuan Guo. 2024-12-03. Automatic State Machine Inference for Binary Protocol Reverse Engineering. https://arxiv.org/abs/2412.02540
Cite the original work for its findings. Save a collection to share your selection of sources.