arXiv subjects

Xiaoguang Mao

Publications and source records attributed to Xiaoguang Mao.

34 records · Page 2Linked to original sources

arXiv2024

DIM: Dynamic Integration of Multimodal Entity Linking with Large Language Model

Our study delves into Multimodal Entity Linking, aligning the mention in multimodal information with entities in knowledge base. Existing methods are still facing challenges like ambiguous entity representations and limited image information utilization. Thus, we propose dynamic entity extraction using ChatGPT, which dynamically extracts entities and enhances datasets. We also propose a method: Dynamically Integrate Multimodal information with knowledge base (DIM), employing the capability of the Large Language Model (LLM) for visual understanding. The LLM, such as BLIP-2, extracts information relevant to entities in the image, which can facilitate improved extraction of entity features and linking them with the dynamic entity representations provided by ChatGPT. The experiments demonstrate that our proposed DIM method outperforms the majority of existing methods on the three original datasets, and achieves state-of-the-art (SOTA) on the dynamically enhanced datasets (Wiki+, Rich+, Diverse+). For reproducibility, our code and collected datasets are released on \url{https://github.com/season1blue/DIM}.

cs.CL↗

arXiv2024

PTA: Enhancing Multimodal Sentiment Analysis through Pipelined Prediction and Translation-based Alignment

Multimodal aspect-based sentiment analysis (MABSA) aims to understand opinions in a granular manner, advancing human-computer interaction and other fields. Traditionally, MABSA methods use a joint prediction approach to identify aspects and sentiments simultaneously. However, we argue that joint models are not always superior. Our analysis shows that joint models struggle to align relevant text tokens with image patches, leading to misalignment and ineffective image utilization. In contrast, a pipeline framework first identifies aspects through MATE (Multimodal Aspect Term Extraction) and then aligns these aspects with image patches for sentiment classification (MASC: Multimodal Aspect-Oriented Sentiment Classification). This method is better suited for multimodal scenarios where effective image use is crucial. We present three key observations: (a) MATE and MASC have different feature requirements, with MATE focusing on token-level features and MASC on sequence-level features; (b) the aspect identified by MATE is crucial for effective image utilization; and (c) images play a trivial role in previous MABSA methods due to high noise. Based on these observations, we propose a pipeline framework that first predicts the aspect and then uses translation-based alignment (TBA) to enhance multimodal semantic consistency for better image utilization. Our method achieves state-of-the-art (SOTA) performance on widely used MABSA datasets Twitter-15 and Twitter-17. This demonstrates the effectiveness of the pipeline approach and its potential to provide valuable insights for future MABSA research. For reproducibility, the code and checkpoint will be released.

cs.CL↗

arXiv2024

DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking

Multimodal entity linking (MEL) aims to utilize multimodal information (usually textual and visual information) to link ambiguous mentions to unambiguous entities in knowledge base. Current methods facing main issues: (1)treating the entire image as input may contain redundant information. (2)the insufficient utilization of entity-related information, such as attributes in images. (3)semantic inconsistency between the entity in knowledge base and its representation. To this end, we propose DWE+ for multimodal entity linking. DWE+ could capture finer semantics and dynamically maintain semantic consistency with entities. This is achieved by three aspects: (a)we introduce a method for extracting fine-grained image features by partitioning the image into multiple local objects. Then, hierarchical contrastive learning is used to further align semantics between coarse-grained information(text and image) and fine-grained (mention and visual objects). (b)we explore ways to extract visual attributes from images to enhance fusion feature such as facial features and identity. (c)we leverage Wikipedia and ChatGPT to capture the entity representation, achieving semantic enrichment from both static and dynamic perspectives, which better reflects the real-world entity semantics. Experiments on Wikimel, Richpedia, and Wikidiverse datasets demonstrate the effectiveness of DWE+ in improving MEL performance. Specifically, we optimize these datasets and achieve state-of-the-art performance on the enhanced datasets. The code and enhanced datasets are released on https://github.com/season1blue/DWET

cs.AI↗

arXiv2023

Enhancing Code Intelligence Tasks with ChatGPT

Pre-trained code models have emerged as crucial tools in various code intelligence tasks. However, their effectiveness depends on the quality of the pre-training dataset, particularly the human reference comments, which serve as a bridge between the programming language and natural language. One significant challenge is that such comments can become inconsistent with the corresponding code as the software evolves. This discrepancy can lead to suboptimal training of the models, decreasing their performances. LLMs have demonstrated superior capabilities in generating high-quality code comments. In light of that, we try to tackle the quality issue of the dataset by harnessing the power of LLMs. Specifically, we raise the question: Can we rebuild the pre-training dataset by substituting the original comments with LLM-generated ones for more effective pre-trained code models? To answer the question, we first conduct a comprehensive evaluation to compare ChatGPT-generated comments with human reference comments. As existing reference-based metrics treat the reference comments as gold standards, we introduce two auxiliary tasks as novel reference-free metrics to assess the quality of comments, i.e., code-comment inconsistency detection and code search. Experimental results show that ChatGPT-generated comments demonstrate superior semantic consistency with the code compared to human references, indicating the potential of utilizing ChatGPT to enhance the quality of the pre-training dataset. We rebuilt the widely used dataset, CodeSearchNet, with ChatGPT-generated comments. Subsequent experiments involve re-pre-training the CodeT5 with our refined dataset.Evaluation results on four generation tasks and one understanding code intelligence tasks show that the model pre-trained by ChatGPT-enhanced data outperforms its counterpart on code summarization, code generation, and code translation tasks.

cs.SE↗

arXiv2023

Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning

Code comment generation aims at generating natural language descriptions for a code snippet to facilitate developers' program comprehension activities. Despite being studied for a long time, a bottleneck for existing approaches is that given a code snippet, they can only generate one comment while developers usually need to know information from diverse perspectives such as what is the functionality of this code snippet and how to use it. To tackle this limitation, this study empirically investigates the feasibility of utilizing large language models (LLMs) to generate comments that can fulfill developers' diverse intents. Our intuition is based on the facts that (1) the code and its pairwise comment are used during the pre-training process of LLMs to build the semantic connection between the natural language and programming language, and (2) comments in the real-world projects, which are collected for the pre-training, usually contain different developers' intents. We thus postulate that the LLMs can already understand the code from different perspectives after the pre-training. Indeed, experiments on two large-scale datasets demonstrate the rationale of our insights: by adopting the in-context learning paradigm and giving adequate prompts to the LLM (e.g., providing it with ten or more examples), the LLM can significantly outperform a state-of-the-art supervised learning approach on generating comments with multiple intents. Results also show that customized strategies for constructing the prompts and post-processing strategies for reranking the results can both boost the LLM's performances, which shed light on future research directions for using LLMs to achieve comment generation.

cs.SE↗

arXiv2023

CCT5: A Code-Change-Oriented Pre-Trained Model

Software is constantly changing, requiring developers to perform several derived tasks in a timely manner, such as writing a description for the intention of the code change, or identifying the defect-prone code changes. Considering that the cost of dealing with these tasks can account for a large proportion (typically around 70 percent) of the total development expenditure, automating such processes will significantly lighten the burdens of developers. To achieve such a target, existing approaches mainly rely on training deep learning models from scratch or fine-tuning existing pretrained models on such tasks, both of which have weaknesses. Specifically, the former uses comparatively small-scale labelled data for training, making it difficult to learn and exploit the domain knowledge of programming language hidden in the large-amount unlabelled code in the wild; the latter is hard to fully leverage the learned knowledge of the pre-trained model, as existing pre-trained models are designed to encode a single code snippet rather than a code change (i.e., the difference between two code snippets). We propose to pre-train a model specially designed for code changes to better support developers in software maintenance. To this end, we first collect a large-scale dataset containing 1.5M+ pairwise data of code changes and commit messages. Based on these data, we curate five different tasks for pre-training, which equip the model with diverse domain knowledge about code changes. We fine-tune the pre-trained model, CCT5, on three widely-studied tasks incurred by code changes and two tasks specific to the code review process. Results show that CCT5 outperforms both conventional deep learning approaches and existing pre-trained models on these tasks.

cs.SE↗

arXiv2022

StyleFlow: Disentangle Latent Representations via Normalizing Flow for Unsupervised Text Style Transfer

Text style transfer aims to alter the style of a sentence while preserving its content. Due to the lack of parallel corpora, most recent work focuses on unsupervised methods and often uses cycle construction to train models. Since cycle construction helps to improve the style transfer ability of the model by rebuilding transferred sentences back to original-style sentences, it brings about a content loss in unsupervised text style transfer tasks. In this paper, we propose a novel disentanglement-based style transfer model StyleFlow to enhance content preservation. Instead of the typical encoder-decoder scheme, StyleFlow can not only conduct the forward process to obtain the output, but also infer to the input through the output. We design an attention-aware coupling layers to disentangle the content representations and the style representations of a sentence. Besides, we propose a data augmentation method based on Normalizing Flow to improve the robustness of the model. Experiment results demonstrate that our model preserves content effectively and achieves the state-of-the-art performance on the most metrics.

cs.CL↗

arXiv2021

Beep: Fine-grained Fix Localization by Learning to Predict Buggy Code Elements

Software Fault Localization refers to the activity of finding code elements (e.g., statements) that are related to a software failure. The state-of-the-art fault localization techniques, however, produce coarse-grained results that can deter manual debugging or mislead automated repair tools. In this work, we focus specifically on the fine-grained identification of code elements (i.e., tokens) that must be changed to fix a buggy program: we refer to it as fix localization. This paper introduces a neural network architecture (named Beep) that builds on AST paths to predict the buggy code element as well as the change action that must be applied to repair a program. Leveraging massive data of bugs and patches within the CoCoNut dataset, we trained a model that was (1) effective in localizing the buggy tokens with the Mean First Rank significantly higher than a statistics based baseline and a machine learning-based baseline, and (2) effective in predicting the repair operators (with the associated buggy code elements) with a Recall@1= 30-45% and the Mean First Rank=7-12 (evaluated by CoCoNut, ManySStuBs4J, and Defects4J datasets). To showcase how fine-grained fix localization can help program repair, we employ it in two repair pipelines where we use either a code completion engine to predict the correct token or a set of heuristics to search for the suitable donor code. A key strength of accurate fix localization for program repair is that it reduces the chance of patch overfitting, a challenge in generate-and-validate automated program repair: both two repair pipelines achieve a correctness ratio of 100%, i.e., all generated patches are found to be correct. Moreover, accurate fix localization helps enhance the efficiency of program repair.

cs.SE↗

arXiv2020

Evolutionary Conflict Checking

During the software evolution, existing features may be adversely affected by new changes, which is well known as regression errors. Maintaining a high-quality test suite is helpful to prevent regression errors, whereas it heavily depends on developers. Continuously augmenting the existing test suite based on the new changes can assist developers in investigating the impact of these new changes. And by comparing the executions of the generated test case on two versions, existing techniques are able to detect some common errors. However, the requirements and oracles on the new changes with existing program behaviors are missing. In addition, the new changes may introduce new bugs when they are not sufficiently examined with other unchanged code, which finally fails to meet developers' real intentions on changes. In this paper, we propose the notion of evolutionary conflict checking to validate new changes. By extracting developers' intention reflected by new changes and transforming the linear evolutionary process into one three-way merge, we detect conflicts between existing behaviors and new changes. Our experimental results indicate that evolutionary conflict checking is able to be applied for guaranteeing software quality after changes.

cs.SE↗

arXiv2020

On the Efficiency of Test Suite based Program Repair: A Systematic Assessment of 16 Automated Repair Systems for Java Programs

Test-based automated program repair has been a prolific field of research in software engineering in the last decade. Many approaches have indeed been proposed, which leverage test suites as a weak, but affordable, approximation to program specifications. Although the literature regularly sets new records on the number of benchmark bugs that can be fixed, several studies increasingly raise concerns about the limitations and biases of state-of-the-art approaches. For example, the correctness of generated patches has been questioned in a number of studies, while other researchers pointed out that evaluation schemes may be misleading with respect to the processing of fault localization results. Nevertheless, there is little work addressing the efficiency of patch generation, with regard to the practicality of program repair. In this paper, we fill this gap in the literature, by providing an extensive review on the efficiency of test suite based program repair. Our objective is to assess the number of generated patch candidates, since this information is correlated to (1) the strategy to traverse the search space efficiently in order to select sensical repair attempts, (2) the strategy to minimize the test effort for identifying a plausible patch, (3) as well as the strategy to prioritize the generation of a correct patch. To that end, we perform a large-scale empirical study on the efficiency, in terms of quantity of generated patch candidates of the 16 open-source repair tools for Java programs. The experiments are carefully conducted under the same fault localization configurations to limit biases.

cs.SE↗

arXiv2020

Automated Regression Unit Test Generation for Program Merges

Merging other branches into the current working branch is common in collaborative software development. However, developers still heavily rely on the textual merge tools to handle the complicated merge tasks. The latent semantic merge conflicts may fail to be detected and degrade the software quality. Regression testing is able to prevent regression faults and has been widely used in real-world software development. However, the merged software may fail to be well examined by rerunning the existing whole test suite. Intuitively, if the test suite fails to cover the changes of different branches at the same time, the merge conflicts would fail to be detected. Recently, it has been proposed to conduct verification on 3-way merges, but this approach does not support even some common cases such as different changes made to different parts of the program. In this paper, we propose an approach of regression unit test generation specifically for checking program merges according to our proposed test oracles. And our general test oracles support us to examine not only 3-way merges, but also 2-way and octopus merges. Considering the conflicts may arise in other locations besides changed methods of the project, we design an algorithm to select UUTs based on the dependency analysis of the whole project. On this basis, we implement a tool called TOM to generate unit tests for Java program merges. We also design the benchmark MCon4J consisting of 389 conflict 3-way merges and 389 conflict octopus merges to facilitate further studies on this topic. The experimental results show that TOM finds 45 conflict 3- way merges and 87 conflicts octopus merges, while the verification based tool fails to work on MCon4J.

cs.SE↗

arXiv2019

How Different Is It Between Machine-Generated and Developer-Provided Patches? An Empirical Study on The Correct Patches Generated by Automated Program Repair Techniques

Background: Over the years, Automated Program Repair (APR) has attracted much attention from both academia and industry since it can reduce the costs in fixing bugs. However, how to assess the patch correctness remains to be an open challenge. Two widely adopted ways to approach this challenge, including manually checking and validating using automated generated tests, are biased (i.e., suffering from subjectivity and low precision respectively). Aim: To address this concern, we propose to conduct an empirical study towards understanding the correct patches that are generated by existing state-of-the-art APR techniques, aiming at providing guidelines for future assessment of patches. Method: To this end, we first present a Literature Review (LR) on the reported correct patches generated by recent techniques on the Defects4J benchmark and collect 177 correct patches after a process of sanity check. We investigate how these machine-generated correct patches achieve semantic equivalence, but syntactic difference compared with developer-provided ones, how these patches distribute in different projects and APR techniques, and how the characteristics of a bug affect the patches generated for it. Results: Our main findings include 1) we do not need to fix bugs exactly like how developers do since we observe that 25.4% (45/177) of the correct patches generated by APR techniques are syntactically different from developer-provided ones; 2) the distribution of machine-generated correct patches diverges for the aspects of Defects4J projects and APR techniques; and 3) APR techniques tend to generate patches that are different from those by developers for bugs with large patch sizes. Conclusion: Our study not only verifies the conclusions from previous studies but also highlights implications for future study towards assessing patch correctness.

cs.SE↗

arXiv2019

Attention Please: Consider Mockito when Evaluating Newly Proposed Automated Program Repair Techniques

Automated program repair (APR) has attracted widespread attention in recent years with substantial techniques being proposed. Meanwhile, a number of benchmarks have been established for evaluating the performances of APR techniques, among which Defects4J is one of the most wildly used benchmark. However, bugs in Mockito, a project augmented in a later-version of Defects4J, do not receive much attention by recent researches. In this paper, we aim at investigating the necessity of considering Mockito bugs when evaluating APR techniques. Our findings show that: 1) Mockito bugs are not more complex for repairing compared with bugs from other projects; 2) the bugs repaired by the state-of-the-art tools share the same repair patterns compared with those patterns required to repair Mockito bugs; however, 3) the state-of-the-art tools perform poorly on Mockito bugs (Nopol can only correctly fix one bug while SimFix and CapGen cannot fix any bug in Mockito even if all the buggy locations have been exposed). We conclude from these results that existing APR techniques may be overfitting to their evaluated subjects and we should consider Mockito, or even more bugs from other projects, when evaluating newly proposed APR techniques. We further find out a unique repair action required to repair Mockito bugs named external package addition. Importing the external packages from the test code associated with the source code is feasible for enlarging the search space and this action can be augmented with existing repair actions to advance existing techniques.

cs.SE↗

arXiv2019

Multi-Location Program Repair Strategies Learned from Past Successful Experience

Automated program repair (APR) has great potential to reduce the effort and time-consumption in software maintenance and becomes a hot topic in software engineering recently with many approaches being proposed. Multi-location program repair has always been a challenge in this field since its complexity in logic and structure. While some approaches do not claim to have the features for solving multi-location bugs, they generate correct patches for these defects in practice. In this paper, we first make an observation on multi-location bugs in Defects4J and divide them into two categories (i.e., similar and relevant multi-location bugs) based on the repair actions in their patches. We then summarize the situation of multi-location bugs in Defects4J fixed by current tools. We analyze the twenty-two patches generated by current tools and propose two feasible strategies for fixing multi-location bugs, illustrating them through two detailed case studies. At last, the experimental results prove the feasibility of our methods with the repair of two bugs that have never been fixed before. By learning from successful experience in the past, this paper points out possible ways ahead for multi-location program repair.

cs.SE↗

arXiv2018

An Initial Step Towards Organ Transplantation Based on GitHub Repository

Organ transplantation, which is the utilization of codes directly related to some specific functionalities to complete ones own program, provides more convenience for developers than traditional component reuse. However, recent techniques are challenged with the lack of organs for transplantation. Hence, we conduct an empirical study on extracting organs from GitHub repository to explore transplantation based on large-scale dataset. We analyze statistics from 12 representative GitHub projects and get the conclusion that 1) there are abundant practical organs existing in commits with add as a key word in the comments; 2) organs in this repository mainly possess four kinds of contents; 3) approximately 70% of the organs are easy-to-transplant. Implementing our transplantation strategy for different kinds of organs, we manually extract 30 organs in three different programming languages, namely Java, Python, and C, and make unit tests for them utilizing four testing tools (two for Java, one for Python, and one for C). At last, we transplant three Java organs into a specific platform for a performance check to verify whether they can work well in the new system. All the 30 organs extracted by our strategy possess good performances in unit test with the highest passing rate reaching 97% and the lowest one still passing 80% and the three Java organs work well in the new system, providing three new functionalities for the host. All the results indicate the feasibility of organ transplantation based on open-source repository, bringing new idea for code reuse.

cs.SE↗

arXiv2013

Counterexample-Preserving Reduction for Symbolic Model Checking

The cost of LTL model checking is highly sensitive to the length of the formula under verification. We observe that, under some specific conditions, the input LTL formula can be reduced to an easier-to-handle one before model checking. In our reduction, these two formulae need not to be logically equivalent, but they share the same counterexample set w.r.t the model. In the case that the model is symbolically represented, the condition enabling such reduction can be detected with a lightweight effort (e.g., with SAT-solving). In this paper, we tentatively name such technique "Counterexample-Preserving Reduction" (CePRe for short), and finally the proposed technquie is experimentally evaluated by adapting NuSMV.

cs.LO↗