Haowen Shen | Searcharxiv

arXiv2024

CIBench: Evaluating Your LLMs with a Code Interpreter Plugin

While LLM-Based agents, which use external tools to solve complex problems, have made significant progress, benchmarking their ability is challenging, thereby hindering a clear understanding of their limitations. In this paper, we propose an interactive evaluation framework, named CIBench, to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks. Our evaluation framework includes an evaluation dataset and two evaluation modes. The evaluation dataset is constructed using an LLM-human cooperative approach and simulates an authentic workflow by leveraging consecutive and interactive IPython sessions. The two evaluation modes assess LLMs' ability with and without human assistance. We conduct extensive experiments to analyze the ability of 24 LLMs on CIBench and provide valuable insights for future LLMs in code interpreter utilization.

cs.CL↗

arXiv2024

ComGPT: Detecting Local Community Structure with Large Language Models

Large Language Models (LLMs), like GPT-3.5-turbo, have demonstrated the ability to understand graph structures and have achieved excellent performance in various graph reasoning tasks, such as node classification. Despite their strong abilities in graph reasoning tasks, they lack specific domain knowledge and have a weaker understanding of community-related graph information, which hinders their capabilities in the community detection task. Moreover, local community detection algorithms based on seed expansion, referred to as seed expansion algorithms, often face several shortcomings, including the seed-dependent problem, community diffusion, and free rider effect. To use LLMs to overcome the above shortcomings, we explore a GPT-guided seed expansion algorithm named ComGPT. ComGPT iteratively selects potential nodes by local modularity from the detected community's neighbors, and subsequently employs LLMs to choose the node from these selected potential nodes to join the detected community. To improve LLMs' understanding of community-related graph information, we propose ComIncident, a graph encoding method that incorporates community knowledge and is designed for the community detection task. Additionally, we design the Node Selection Guide (NSG) prompt to enhance LLMs' understanding of community characteristics. Experimental results demonstrate that ComGPT outperforms the baselines, thereby confirming the effectiveness of the ComIncident and the NSG prompt.

cs.SI↗