arXiv subjects

Bin Xu

Publications and source records attributed to Bin Xu.

At least 109 records · Page 6Linked to original sources

arXiv2025

LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models

Existing Large Vision-Language Models (LVLMs) can process inputs with context lengths up to 128k visual and text tokens, yet they struggle to generate coherent outputs beyond 1,000 words. We find that the primary limitation is the absence of long output examples during supervised fine-tuning (SFT). To tackle this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158 examples, each with multiple input images, an instruction, and corresponding outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that maintain high-fidelity to the input images, we employ Direct Preference Optimization (DPO) to the SFT model. Given the high cost of collecting human feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which breaks long outputs into segments and uses iterative corrections to form preference pairs with the original outputs. Additionally, we develop MMLongBench-Write, a benchmark featuring six tasks to evaluate the long-generation capabilities of VLMs. Our 7B parameter model, trained with LongWriter-V-22k and IterDPO, achieves impressive performance on this benchmark, outperforming larger proprietary models like GPT-4o. Code and data: https://github.com/THU-KEG/LongWriter-V

cs.CV↗

arXiv2025

Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction

The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evaluation benchmarks have been devoted to validating pairwise matching correctness, ignoring the crucial measurement of robustness. In this paper, we present the first benchmark that simulates the evaluation of open information extraction models in the real world, where the syntactic and expressive distributions under the same knowledge meaning may drift variously. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique that consists of sentences with structured knowledge of the same meaning but with different syntactic and expressive forms. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques. We perform experiments on typical models published in the last decade as well as a popular large language model, the results show that the existing successful models exhibit a frustrating degradation, with a maximum drop of 23.43 F1 score. Our resources and code are available at https://github.com/qijimrc/ROBUST.

cs.CL↗

arXiv2025

Generating customized prompts for Zero-Shot Rare Event Medical Image Classification using LLM

Rare events, due to their infrequent occurrences, do not have much data, and hence deep learning techniques fail in estimating the distribution for such data. Open-vocabulary models represent an innovative approach to image classification. Unlike traditional models, these models classify images into any set of categories specified with natural language prompts during inference. These prompts usually comprise manually crafted templates (e.g., 'a photo of a {}') that are filled in with the names of each category. This paper introduces a simple yet effective method for generating highly accurate and contextually descriptive prompts containing discriminative characteristics. Rare event detection, especially in medicine, is more challenging due to low inter-class and high intra-class variability. To address these, we propose a novel approach that uses domain-specific expert knowledge on rare events to generate customized and contextually relevant prompts, which are then used by large language models for image classification. Our zero-shot, privacy-preserving method enhances rare event classification without additional training, outperforming state-of-the-art techniques.

cs.CV↗

arXiv2025

A note On the existence of solutions to Hitchin's self-duality equations

In 1987, Hitchin introduced the self-duality equations on rank-2 complex vector bundles over compact Riemann surfaces with genus greater than one as a reduction of the Yang-Mills equation and established the existence of solutions to these equations starting from a Higgs stable bundle. In this paper, we fill in some technical details in Hitchin's original proof by the following three steps. First, we reduce the existence of a solution of class $L_1^2$ to minimizing the energy functional within a Higgs stable orbit of the $L_2^2$ complex gauge group action. Second, using this transformation, we obtain a solution of class $L_1^2$ in this orbit. These two steps primarily follow Hitchin's original approach. Finally, using the Coulomb gauge, we construct a smooth solution by applying an $L_2^2$ unitary gauge transformation to the $L_1^2$ solution constructed previously. This last step provides additional technical details to Hitchin's original proof.

math.DG↗

arXiv2025

The moduli space of HCMU surfaces

HCMU surfaces are compact Riemann surfaces equipped with an extremal Kähler metric and a finite number of singularities. Research on these surfaces was initiated by E. Calabi and X.-X. Chen over thirty years ago. We provide a detailed description of the geometric structure of HCMU surfaces, building on the classical football decomposition introduced by Chen-Chen-Wu. From this perspective, most HCMU surfaces can be uniquely described by a set of data that includes both discrete topological information and continuous geometric parameters. This data representation is effective for studying the moduli space of HCMU surfaces with specified genus and conical angles, suggesting a topological approach to this topic. As a first application, we present a unified proof of the angle constraints on HCMU surfaces. Using the same approach, we establish an existence theorem for HCMU surfaces of any genus with a single conical point, which is also a saddle point. Finally, we determine the dimension of the moduli space, defined as the number of independent continuous parameters. This is achieved by examining several geometric deformations of HCMU surfaces and the various relationships between the quantities in the data set representation.

math.GT↗

arXiv2024

CogAgent: A Visual Language Model for GUI Agents

People are spending an enormous amount of time on digital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks like writing emails, but struggle to understand and interact with GUIs, thus limiting their potential to increase automation levels. In this paper, we introduce CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation. By utilizing both low-resolution and high-resolution image encoders, CogAgent supports input at a resolution of 1120*1120, enabling it to recognize tiny page elements and text. As a generalist visual language model, CogAgent achieves the state of the art on five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using only screenshots as input, outperforms LLM-based methods that consume extracted HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW, advancing the state of the art. The model and codes are available at https://github.com/THUDM/CogVLM, with a new version of CogAgent-9B-20241220 available at https://github.com/THUDM/CogAgent.

cs.CV↗

arXiv2024

MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Model

Large language models (LLMs) have demonstrated significant capabilities in mathematical reasoning, particularly with text-based mathematical problems. However, current multi-modal large language models (MLLMs), especially those specialized in mathematics, tend to focus predominantly on solving geometric problems but ignore the diversity of visual information available in other areas of mathematics. Moreover, the geometric information for these specialized mathematical MLLMs is derived from several public datasets, which are typically limited in diversity and complexity. To address these limitations, we aim to construct a fine-tuning dataset named MathVL, and develop a series of specialized mathematical MLLMs termed MathGLM-Vision by conducting Supervised Fine-Tuning (SFT) on MathVL with various parameter-scale backbones. To extensively evaluate the effectiveness of MathGLM-Vision, we conduct experiments on several public benchmarks and our curated MathVL-test consisting of 2,000 problems. Experimental results demonstrate that MathGLM-Vision achieves significant improvements compared with some existing models, including backbone models and open-source mathematical MLLMs. These findings indicate the importance of diversity dataset in enhancing the mathematical reasoning abilities of MLLMs.

cs.CL↗

arXiv2024

VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning

Multi-modal large language models (MLLMs) have demonstrated promising capabilities across various tasks by integrating textual and visual information to achieve visual understanding in complex scenarios. Despite the availability of several benchmarks aims to evaluating MLLMs in tasks from visual question answering to complex problem-solving, most focus predominantly on mathematics or general visual understanding tasks. This reveals a critical gap in current benchmarks, which often overlook the inclusion of other key scientific disciplines such as physics and chemistry. To address this gap, we meticulously construct a comprehensive benchmark, named VisScience, which is utilized to assess the multi-modal scientific reasoning across the three disciplines of mathematics, physics, and chemistry. This benchmark comprises 3,000 questions drawn from K12 education - spanning elementary school through high school - equally distributed across three disciplines, with 1,000 questions per discipline. The questions within VisScience span 21 distinct subjects and are categorized into five difficulty levels, offering a broad spectrum of topics within each discipline. With VisScience, we present a detailed evaluation of the performance of 25 representative MLLMs in scientific reasoning. Experimental results demonstrate that closed-source MLLMs generally outperform open-source models. The best performance observed include a 53.4\% accuracy in mathematics by Claude3.5-Sonnet, 38.2\% in physics by GPT-4o, and 47.0\% in chemistry by Gemini-1.5-Pro. These results underscore the strengths and limitations of MLLMs, suggesting areas for future improvement and highlighting the importance of developing models that can effectively handle the diverse demands of multi-modal scientific reasoning.

cs.AI↗

arXiv2024

Eliminating Incoherent Noise: A Coherent Quantum Approach in Multi-Sensor Dark Matter Detection

We propose a novel dark matter detection scheme by leveraging quantum coherence across a network of multiple quantum sensors. This method effectively eliminates incoherent background noise, thereby significantly enhancing detection sensitivity. This is achieved by performing a series of basis transformation operations, allowing the coherent signal to be expressed as a combination of sensor population measurements without introducing background noise. We present a comprehensive analytical analysis and complement it with practical numerical simulations. These demonstrations reveal that signal strength is enhanced by the square of the number of sensors, while noise, primarily due to operational infidelity rather than background fluctuations, increases only linearly with the number of sensors. Our approach paves the way for next-generation dark matter searches that optimally utilize an advanced network of sensors and quantum technologies.

hep-ph↗

arXiv2024

ADELIE: Aligning Large Language Models on Information Extraction

Large language models (LLMs) usually fall short on information extraction (IE) tasks and struggle to follow the complex instructions of IE tasks. This primarily arises from LLMs not being aligned with humans, as mainstream alignment datasets typically do not include IE data. In this paper, we introduce ADELIE (Aligning large language moDELs on Information Extraction), an aligned LLM that effectively solves various IE tasks, including closed IE, open IE, and on-demand IE. We first collect and construct a high-quality alignment corpus IEInstruct for IE. Then we train ADELIE_SFT using instruction tuning on IEInstruct. We further train ADELIE_SFT with direct preference optimization (DPO) objective, resulting in ADELIE_DPO. Extensive experiments on various held-out IE datasets demonstrate that our models (ADELIE_SFT and ADELIE_DPO) achieve state-of-the-art (SoTA) performance among open-source models. We further explore the general capabilities of ADELIE, and experimental results reveal that their general capabilities do not exhibit a noticeable decline. We will release the code, data, and models to facilitate further research.

cs.CL↗

arXiv2024

Stochastic evolution elasto-plastic modeling of a metallic glass

This paper develops a general data-driven approach to stochastic elastoplastic modelling that leverages atomistic simulation data directly rather than by fitting parameters. The approach is developed in the context of metallic glasses, which present inherent complexities due to their disordered structure. By harvesting statistics from simulated metallic glass shear response histories, the material state is mapped onto a two-dimensional state space consisting of the shear stress and the inelastic contribution to the potential energy. The resulting elastoplastic model is intrinsically stochastic and represented as a non-deterministic dynamical map. The state space statistics provide insights into the deformation physics of metallic glasses, revealing that two state variables are sufficient to describe the main features of the elastoplastic response. In this two-dimensional state space, the gradually quenched metallic glass rejuvenates during the initial quasi-elastic shearing, ultimately reaching a steady state that fluctuates about a fixed point in the state space as rejuvenation and aging balance.

cond-mat.stat-mech↗

arXiv2024

Existence and non-uniqueness of cone spherical metrics with prescribed singularities on a compact Riemann surface with positive genus

Cone spherical metrics, defined on compact Riemann surfaces, are conformal metrics with constant curvature one and finitely many cone singularities. Such a metric is termed \textit{reducible} if a developing map of the metric has monodromy in ${\rm U(1)}$, and \textit{irreducible} otherwise. Utilizing the polystable extensions of two line bundles on a compact Riemann surface $X$ with genus $g_X>0$, we establish the following three primary results concerning these metrics with cone angles in $2π{\mathbb Z}_{>1}$: \begin{itemize} \item[(1)] Given an effective divisor $D$ with an odd degree surpassing $2g_X$ on $X$, we find the existence of an effective divisor $D'$ in the complete linear system $|D|$ that can be represented by at least two distinct irreducible cone spherical metrics on $X$. \item[(2)] For a generic effective divisor $D$ with an even degree and $°D\geq 6g_X-2$ on $X$, we can identify an arcwise connected Borel subset in $|D|$ that demonstrates a Hausdorff dimension of no less than $\big(°D-4g_{X}+2\big)$. Within this subset, each divisor $D'$ can be distinctly represented by a family of reducible metrics, defined by a single real parameter. \item[(3)] For an effective divisor $D$ with $°D=2$ on an elliptic curve, we can identify a Borel subset in $|D|$ that is arcwise connected, showcasing a Hausdorff dimension of one. Within this subset, each divisor $D'$ can be distinctly represented by a family of reducible metrics, defined by a single real parameter.

math.DG↗

arXiv2024

Geometric structure and existence of reducible spherical conical metrics

A conformal metric ${\rm d}s^{2}$ with finitely many conical singularities of constant Gaussian curvature $K=1$ on a compact Riemann surface is referred to as a spherical conical metric. When the associated monodromy group of ${\rm d}s^{2}$ is diagonalizable, we refer to ${\rm d}s^{2}$ as a reducible spherical conical metric. The simplest case of a reducible spherical conical metric is a `football', which denotes a 2-sphere with a spherical conical metric that has precisely two singularities separated by a distance of $π$. This study delves into the intrinsic geometric structure and existence of reducible spherical conical metrics on compact Riemann surfaces. We demonstrate that any such spherical surface can be divided into a finite number of pieces by cutting along a set of suitable geodesics, which connect the conical singularities and some smooth points of the metric. Especially, each piece is isometric to a portion obtained by cutting a football along a geodesic that joins the two conical singularities. As an application, an angle condition for the existence of such a metric is presented. Most notably, our study demonstrates the existence of a reducible spherical conical metric where all the saddle points of a Morse function are located on the same geodesic.

math.DG↗

arXiv2024

CogVLM2: Visual Language Models for Image and Video Understanding

Beginning with VisualGLM and CogVLM, we are continuously exploring VLMs in pursuit of enhanced vision-language fusion, efficient higher-resolution architecture, and broader modalities and applications. Here we propose the CogVLM2 family, a new generation of visual language models for image and video understanding including CogVLM2, CogVLM2-Video and GLM-4V. As an image understanding model, CogVLM2 inherits the visual expert architecture with improved training recipes in both pre-training and post-training stages, supporting input resolution up to $1344 \times 1344$ pixels. As a video understanding model, CogVLM2-Video integrates multi-frame input with timestamps and proposes automated temporal grounding data construction. Notably, CogVLM2 family has achieved state-of-the-art results on benchmarks like MMBench, MM-Vet, TextVQA, MVBench and VCGBench. All models are open-sourced in https://github.com/THUDM/CogVLM2 and https://github.com/THUDM/GLM-4, contributing to the advancement of the field.

cs.CV↗

arXiv2024

ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools

We introduce ChatGLM, an evolving family of large language models that we have been developing over time. This report primarily focuses on the GLM-4 language series, which includes GLM-4, GLM-4-Air, and GLM-4-9B. They represent our most capable models that are trained with all the insights and lessons gained from the preceding three generations of ChatGLM. To date, the GLM-4 models are pre-trained on ten trillions of tokens mostly in Chinese and English, along with a small set of corpus from 24 languages, and aligned primarily for Chinese and English usage. The high-quality alignment is achieved via a multi-stage post-training process, which involves supervised fine-tuning and learning from human feedback. Evaluations show that GLM-4 1) closely rivals or outperforms GPT-4 in terms of general metrics such as MMLU, GSM8K, MATH, BBH, GPQA, and HumanEval, 2) gets close to GPT-4-Turbo in instruction following as measured by IFEval, 3) matches GPT-4 Turbo (128K) and Claude 3 for long context tasks, and 4) outperforms GPT-4 in Chinese alignments as measured by AlignBench. The GLM-4 All Tools model is further aligned to understand user intent and autonomously decide when and which tool(s) touse -- including web browser, Python interpreter, text-to-image model, and user-defined functions -- to effectively complete complex tasks. In practical applications, it matches and even surpasses GPT-4 All Tools in tasks like accessing online information via web browsing and solving math problems using Python interpreter. Over the course, we have open-sourced a series of models, including ChatGLM-6B (three generations), GLM-4-9B (128K, 1M), GLM-4V-9B, WebGLM, and CodeGeeX, attracting over 10 million downloads on Hugging face in the year 2023 alone. The open models can be accessed through https://github.com/THUDM and https://huggingface.co/THUDM.

cs.CL↗

arXiv2024

MAVEN-Fact: A Large-scale Event Factuality Detection Dataset

Event Factuality Detection (EFD) task determines the factuality of textual events, i.e., classifying whether an event is a fact, possibility, or impossibility, which is essential for faithfully understanding and utilizing event knowledge. However, due to the lack of high-quality large-scale data, event factuality detection is under-explored in event understanding research, which limits the development of EFD community. To address these issues and provide faithful event understanding, we introduce MAVEN-Fact, a large-scale and high-quality EFD dataset based on the MAVEN dataset. MAVEN-Fact includes factuality annotations of 112,276 events, making it the largest EFD dataset. Extensive experiments demonstrate that MAVEN-Fact is challenging for both conventional fine-tuned models and large language models (LLMs). Thanks to the comprehensive annotations of event arguments and relations in MAVEN, MAVEN-Fact also supports some further analyses and we find that adopting event arguments and relations helps in event factuality detection for fine-tuned models but does not benefit LLMs. Furthermore, we preliminarily study an application case of event factuality detection and find it helps in mitigating event-related hallucination in LLMs. Our dataset and codes can be obtained from \url{https://github.com/lcy2723/MAVEN-FACT}

cs.CL↗

arXiv2024

HPHS: Hierarchical Planning based on Hybrid Frontier Sampling for Unknown Environments Exploration

Rapid sampling from the environment to acquire available frontier points and timely incorporating them into subsequent planning to reduce fragmented regions are critical to improve the efficiency of autonomous exploration. We propose HPHS, a fast and effective method for the autonomous exploration of unknown environments. In this work, we efficiently sample frontier points directly from the LiDAR data and the local map around the robot, while exploiting a hierarchical planning strategy to provide the robot with a global perspective. The hierarchical planning framework divides the updated environment into multiple subregions and arranges the order of access to them by considering the overall revenue of the global path. The combination of the hybrid frontier sampling method and hierarchical planning strategy reduces the complexity of the planning problem and mitigates the issue of region remnants during the exploration process. Detailed simulation and real-world experiments demonstrate the effectiveness and efficiency of our approach in various aspects. The source code will be released to benefit the further research.

cs.RO↗

arXiv2024

KoLA: Carefully Benchmarking World Knowledge of Large Language Models

The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we carefully design three crucial factors: (1) For \textbf{ability modeling}, we mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering $19$ tasks. (2) For \textbf{data}, to ensure fair comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, aiming to evaluate the capacity to handle unseen data and evolving knowledge. (3) For \textbf{evaluation criteria}, we adopt a contrastive system, including overall standard scores for better numerical comparability across tasks and models and a unique self-contrast metric for automatically evaluating knowledge-creating ability. We evaluate $28$ open-source and commercial LLMs and obtain some intriguing findings. The KoLA dataset and open-participation leaderboard are publicly released at https://kola.xlore.cn and will be continuously updated to provide references for developing LLMs and knowledge-related systems.

cs.CL↗