arXiv subjects

Wenjia Song

Publications and source records attributed to Wenjia Song.

5 recordsLinked to original sources

arXiv2026

How Can ChatGPT Support Human Security Testers to Help Mitigate Supply Chain Attacks?

Developers often build software on top of third-party libraries (Libs) to improve productivity, but these libraries may contain vulnerabilities that enable supply chain attacks. Existing tools detect vulnerable dependencies, yet developers often distrust their reports without concrete exploit evidence. Manually crafting such demonstrations is costly, and tool support is lacking. To help developers enhance software security, in this study, we systematically explored the usage of a large language model (LLM) --ChatGPT-4.0--to generate security tests, which unit tests demonstrate how vulnerable library dependencies facilitate the supply chain attacks to given Apps. In our exploration, we defined prompt templates to take in the various vulnerability-relevant information we manually collected, and generated prompts from those templates to query ChatGPT for security test generation. We found that ChatGPT-generated tests demonstrated 24 pieces of evidence or proof of vulnerability for 49 Apps. To assess the consistency of test generation, we also evaluated another five state-of-the-art LLMs. All the models generated security tests for at least 17 cases that successfully demonstrate the vulnerabilities. We filed six reports for the newly revealed vulnerabilities in Apps, and got four Common Vulnerability Entries (CVEs) assigned. Our use of ChatGPT outperformed two state-of-the-art security test generators (TRANSFER and SIEGE), by generating a lot more tests and achieving more attacks.

cs.CR↗

arXiv2025

CognixShield: PoV-Guided Vulnerable API Usage Detection in Large Codebases via LLMs

The integration of open-source libraries in Java development introduces severe security risks through vulnerable APIs. Existing program analysis and deep learning tools face the challenge of capturing inter-procedural vulnerability semantics at scale. While LLMs show promise for semantic reasoning, they cannot handle large codebases because of context limits and a lack of vulnerability-specific understanding needed to determine exploitability. We present CognixShield, an LLM-powered framework for detecting vulnerable API usage through three core components. First, semantic-preserving AST-based fragmentation partitions large codebases while maintaining syntactic completeness within LLM context-window limits. Second, vulnerability-aware multi-agent RAG traces relevant program context across these fragments, iteratively assembling security-critical context spanning functions and files. Third, PoV-guided semantic reasoning leverages Proof-of-Vulnerability tests that encode precise triggering conditions and exploitation mechanics for vulnerability determination. CognixShield achieves 84% precision, 95% recall, 84% accuracy, and an 89% F1-score on 57 real-world Java applications, outperforming state-of-the-art tools. Our results show that vulnerability detection requires specialized architectural innovations beyond generic LLM applications.

cs.SE↗

arXiv2024

Crypto-Ransomware and Their Defenses: In-depth Behavioral Characterization, Discussion of Deployability, and New Insights

Crypto-ransomware has caused an unprecedented scope of impact in recent years with an evolving level of sophistication. An extensive range of studies have been on defending against ransomware and reviewing the efficacy of various protections. However, for practical defenses, deployability holds equal significance as detection accuracy. Therefore, in this study, we review 117 published ransomware defense works, categorize them by the level they are implemented at, and discuss the deployability. API-based solutions are easy to deploy and most existing works focus on machine learning-based classification. To provide more insights, we quantitively characterize the runtime behaviors of real-world ransomware samples. Based on our experimental findings, we present a possible future detection direction with our consistency analysis and API-contrast-based refinement. Moreover, we experimentally evaluate various commercial defenses and identify the security gaps. Our findings help the field understand the deployability of ransomware defenses and create more effective, practical solutions.

cs.CR↗

arXiv2024

Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection

With the expanding application of Large Language Models (LLMs) in various domains, it becomes imperative to comprehensively investigate their unforeseen behaviors and consequent outcomes. In this study, we introduce and systematically explore the phenomenon of "glitch tokens", which are anomalous tokens produced by established tokenizers and could potentially compromise the models' quality of response. Specifically, we experiment on seven top popular LLMs utilizing three distinct tokenizers and involving a totally of 182,517 tokens. We present categorizations of the identified glitch tokens and symptoms exhibited by LLMs when interacting with glitch tokens. Based on our observation that glitch tokens tend to cluster in the embedding space, we propose GlitchHunter, a novel iterative clustering-based technique, for efficient glitch token detection. The evaluation shows that our approach notably outperforms three baseline methods on eight open-source LLMs. To the best of our knowledge, we present the first comprehensive study on glitch tokens. Our new detection further provides valuable insights into mitigating tokenization-related errors in LLMs.

cs.CL↗

arXiv2021

Embedding Code Contexts for Cryptographic API Suggestion:New Methodologies and Comparisons

Despite recent research efforts, the vision of automatic code generation through API recommendation has not been realized. Accuracy and expressiveness challenges of API recommendation needs to be systematically addressed. We present a new neural network-based approach, Multi-HyLSTM for API recommendation --targeting cryptography-related code. Multi-HyLSTM leverages program analysis to guide the API embedding and recommendation. By analyzing the data dependence paths of API methods, we train embedding and specialize a multi-path neural network architecture for API recommendation tasks that accurately predict the next API method call. We address two previously unreported programming language-specific challenges, differentiating functionally similar APIs and capturing low-frequency long-range influences. Our results confirm the effectiveness of our design choices, including program-analysis-guided embedding, multi-path code suggestion architecture, and low-frequency long-range-enhanced sequence learning, with high accuracy on top-1 recommendations. We achieve a top-1 accuracy of 91.41% compared with 77.44% from the state-of-the-art tool SLANG. In an analysis of 245 test cases, compared with the commercial tool Codota, we achieve a top-1 recommendation accuracy of 88.98%, which is significantly better than Codota's accuracy of 64.90%. We publish our data and code as a large Java cryptographic code dataset.

cs.SE↗