arXiv subjects

Kaiming Xue

Publications and source records attributed to Kaiming Xue.

3 recordsLinked to original sources

arXiv2026

Using OAI Overlay to Enhance REST API Fuzzing

REST APIs are widely used in industry. Therefore, a lot of research has been focused on how to automatically generate test cases for REST APIs, with few different open-source fuzzers existing in the literature. For a thorough testing, especially in black-box scenarios, just relying on the information provided in the OpenAPI schemas is not enough. Testers typically need to provide extra input data to help steer the fuzzers in the right direction. Dedicated formats specific to each different fuzzer would work, but they would create a vendor lock-in, as well as increasing cognitive load. The OpenAPI Initiative (OAI) standard Overlay might be a solution to this problem. Such standard enables to define transformations on the OpenAPI schemas, where testers can provide input data in Overlay files where such data is provided as ``examples'' entries. In this paper, we have extended the state-of-the-art fuzzer EvoMaster to support Overlay files natively. Experiments are carried out in industry on five APIs from five enterprises from around the world (e.g., Belgium, China, Germany and T\"urkiye), including two Fortune500 enterprises as well as a 3-man startup. Our industrial results show that Overlay is a viable solution to better enable black-box fuzzing of REST APIs in industry.

cs.SE↗

arXiv2023

White-box Fuzzing RPC-based APIs with EvoMaster: An Industrial Case Study

Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over a network. RPC is widely applied in industry for building large-scale distributed systems, such as Microservices. Modern RPC frameworks include for example Thrift, gRPC, SOFARPC and Dubbo. Testing such systems is very challenging, due to the complexity of distributed systems and various RPC frameworks the system could employ. To the best of our knowledge, there does not exist any tool or solution that could enable automated testing of modern RPC-based services. To fill this gap, in this paper we propose the first approach in the literature, together with an open-source tool, for white-box fuzzing modern RPC-based APIs with search. To assess our novel approach, we conducted an empirical study with two artificial and four industrial APIs selected by our industrial partner. The tool has been integrated into a real industrial pipeline, and could be applied to real industrial development process for fuzzing RPC-based APIs. To further demonstrate its effectiveness and application in industrial settings, we also report results of employing our tool for fuzzing another 50 industrial APIs autonomously conducted by our industrial partner in their testing processes. Results show that our novel approach is capable of enabling automated test case generation for industrial RPC-based APIs (i.e., two artificial and 54 industrial). We also compared with a simple grey-box technique and existing manually written tests. Our white-box solution achieves significant improvements on code coverage. Regarding fault detection, by conducting a careful review with our industrial partner of the tests generated by our novel approach in the selected four industrial APIs, a total of 41 real faults were identified, which have now been fixed. Another 8,377 detected faults are currently under investigation.

cs.SE↗

arXiv2022

Fuzzing Microservices: A Series of User Studies in Industry on Industrial Systems with EvoMaster

With several microservice architectures comprising of thousands of web services, used to serve 630 million customers, companies like Meituan face several challenges in the verification and validation of their software. This paper reports on our experience of integrating EvoMaster (a search-based white-box fuzzer) in the testing processes at Meituan over almost 2 years. Two user studies were carried out in 2021 and in 2023 to evaluate two versions of EvoMaster, respectively, in tackling the test generation for industrial web services which are parts of a large e-commerce microservice system. The two user studies involve in total 321,131 lines of code from five APIs and 27 industrial participants at Meituan. Questionnaires and interviews were carried out in both user studies with employees at Meituan. The two user studies demonstrate clear advantages of EvoMaster (i.e., code coverage and fault detection) and the urgent need to have such a fuzzer in industrial microservices testing. To study how these results could generalize, a follow up user study was done in 2024 with five engineers in the five different companies. Our results show that, besides their clear usefulness, there are still many critical challenges that the research community needs to investigate to improve performance further.

cs.SE↗