From Retrieved Evidence to Security Outcomes: A Data-Driven Analysis of Java Security API Misuse in LLM-Generated Code
The misuse of Java security APIs is a serious security problem in software development. Large language models can generate compilable code, while still misusing cryptographic and secure-communication APIs. We investigate whether external security evidence can reduce this risk and how retriever choice and evidence source shape the resulting programs. Our study evaluates the GPT-5.5 model and Llama-3.3-70B-Instruct model on 12 Java Cryptography Architecture (JCA) and Java Secure Socket Extension (JSSE) security functionalities under 11 prompt and evidence conditions, yielding 23,760 generated programs. We measure program validity, detected Java security API misuse, and the top-5 evidence selected by four full-corpus retrievers. Under task-only prompting, 58.5% of valid Llama-3.3-70B-Instruct-generated programs and 36.6% of valid GPT-5.5-generated programs contain detected security API misuse. Dense full-corpus retrieval reduces conditional misuse among valid programs by 29.0--32.7 and 22.4--31.0 percentage points, respectively. The retrievers show preferences for substantially different evidence types. BM25 and the OpenAI embedding retrievers favor official guidance, the Qwen embedding retriever favors code examples, and the Jina embedding retriever selects a more heterogeneous mixture. Under source-restricted retrieval, code examples yield the lowest observed conditional misuse for the Llama-3.3-70B-Instruct model, while explicit misuse patterns produce zero detected misuse cases for known benchmark patterns among 1,016 valid programs generated by the GPT-5.5 model, with validity reduced to 94.1%. External evidence can substantially reduce detected misuse in the evaluated settings, and the results motivate actionable evidence selection and post-generation validation.