arXiv subjects

Tong Zhu

Publications and source records attributed to Tong Zhu.

At least 73 records · Page 4Linked to original sources

arXiv2024

ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM

Large language models (LLMs) have achieved impressive advancements across numerous disciplines, yet the critical issue of knowledge conflicts, a major source of hallucinations, has rarely been studied. Only a few research explored the conflicts between the inherent knowledge of LLMs and the retrieved contextual knowledge. However, a thorough assessment of knowledge conflict in LLMs is still missing. Motivated by this research gap, we present ConflictBank, the first comprehensive benchmark developed to systematically evaluate knowledge conflicts from three aspects: (i) conflicts encountered in retrieved knowledge, (ii) conflicts within the models' encoded knowledge, and (iii) the interplay between these conflict forms. Our investigation delves into four model families and twelve LLM instances, meticulously analyzing conflicts stemming from misinformation, temporal discrepancies, and semantic divergences. Based on our proposed novel construction framework, we create 7,453,853 claim-evidence pairs and 553,117 QA pairs. We present numerous findings on model scale, conflict causes, and conflict types. We hope our ConflictBank benchmark will help the community better understand model behavior in conflicts and develop more reliable LLMs.

cs.CL↗

arXiv2024

Timo: Towards Better Temporal Reasoning for Language Models

Reasoning about time is essential for Large Language Models (LLMs) to understand the world. Previous works focus on solving specific tasks, primarily on time-sensitive question answering. While these methods have proven effective, they cannot generalize to a wider spectrum of temporal reasoning tasks. Therefore, we propose a crucial question: Can we build a universal framework to handle a variety of temporal reasoning tasks? To that end, we systematically study 38 temporal reasoning tasks. Based on the observation that 19 tasks are directly related to mathematics, we first leverage the available mathematical dataset to set a solid foundation for temporal reasoning. However, the in-depth study indicates that focusing solely on mathematical enhancement falls short of addressing pure temporal reasoning tasks. To mitigate this limitation, we propose a simple but effective self-critic temporal optimization method to enhance the model's temporal reasoning capabilities without sacrificing general task abilities. Finally, we develop Timo, a model designed to excel in temporal reasoning at the 7B and 13B scales. Notably, Timo outperforms the counterpart LLMs by 10.0 and 7.6 in average accuracy scores and achieves the new state-of-the-art (SOTA) performance of comparable size. Extensive experiments further validate our framework's effectiveness and its generalization across diverse temporal tasks. The code is available at https://github.com/zhaochen0110/Timo.

cs.CL↗

arXiv2024

DPA-2: a large atomic model as a multi-task learner

The rapid advancements in artificial intelligence (AI) are catalyzing transformative changes in atomic modeling, simulation, and design. AI-driven potential energy models have demonstrated the capability to conduct large-scale, long-duration simulations with the accuracy of ab initio electronic structure methods. However, the model generation process remains a bottleneck for large-scale applications. We propose a shift towards a model-centric ecosystem, wherein a large atomic model (LAM), pre-trained across multiple disciplines, can be efficiently fine-tuned and distilled for various downstream tasks, thereby establishing a new framework for molecular modeling. In this study, we introduce the DPA-2 architecture as a prototype for LAMs. Pre-trained on a diverse array of chemical and materials systems using a multi-task approach, DPA-2 demonstrates superior generalization capabilities across multiple downstream tasks compared to the traditional single-task pre-training and fine-tuning methodologies. Our approach sets the stage for the development and broad application of LAMs in molecular and materials simulation research.

physics.chem-ph↗

arXiv2024

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale setting still suffers from data-hungry and instability problems. Motivated by this limit, we investigate building MoE models from existing dense large language models. Specifically, based on the well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert Construction, which partitions the parameters of original Feed-Forward Networks (FFNs) into multiple experts; (2) Continual Pre-training, which further trains the transformed MoE model and additional gate networks. In this paper, we comprehensively explore different methods for expert construction and various data sampling strategies for continual pre-training. After these stages, our LLaMA-MoE models could maintain language abilities and route the input tokens to specific experts with part of the parameters activated. Empirically, by training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense models that contain similar activation parameters. The source codes and models are available at https://github.com/pjlab-sys4nlp/llama-moe .

cs.CL↗

arXiv2024

Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts

Mixture-of-Experts (MoE) models have shown remarkable capability in instruction tuning, especially when the number of tasks scales. However, previous methods simply merge all training tasks (e.g. creative writing, coding, and mathematics) and apply fixed sampling weights, without considering the importance of different tasks as the model training state changes. In this way, the most helpful data cannot be effectively distinguished, leading to suboptimal model performance. To reduce the potential redundancies of datasets, we make the first attempt and propose a novel dynamic data mixture for MoE instruction tuning. Specifically, inspired by MoE's token routing preference, we build dataset-level representations and then capture the subtle differences among datasets. Finally, we propose to dynamically adjust the sampling weight of datasets by their inter-redundancies, thus maximizing global performance under a limited training budget. The experimental results on two MoE models demonstrate the effectiveness of our approach on both downstream knowledge \& reasoning tasks and open-ended queries. Code and models are available at https://github.com/Spico197/MoE-SFT .

cs.CL↗

arXiv2024

Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning?

Temporal reasoning is fundamental for large language models (LLMs) to comprehend the world. Current temporal reasoning datasets are limited to questions about single or isolated events, falling short in mirroring the realistic temporal characteristics involving concurrent nature and intricate temporal interconnections. In this paper, we introduce CoTempQA, a comprehensive co-temporal Question Answering (QA) benchmark containing four co-temporal scenarios (Equal, Overlap, During, Mix) with 4,748 samples for evaluating the co-temporal comprehension and reasoning abilities of LLMs. Our extensive experiments reveal a significant gap between the performance of current LLMs and human-level reasoning on CoTempQA tasks. Even when enhanced with Chain of Thought (CoT) methodologies, models consistently struggle with our task. In our preliminary exploration, we discovered that mathematical reasoning plays a significant role in handling co-temporal events and proposed a strategy to boost LLMs' co-temporal reasoning from a mathematical perspective. We hope that our CoTempQA datasets will encourage further advancements in improving the co-temporal reasoning capabilities of LLMs. Our code is available at https://github.com/zhaochen0110/Cotempqa.

cs.CL↗

arXiv2024

Probing Language Models for Pre-training Data Detection

Large Language Models (LLMs) have shown their impressive capabilities, while also raising concerns about the data contamination problems due to privacy issues and leakage of benchmark datasets in the pre-training phase. Therefore, it is vital to detect the contamination by checking whether an LLM has been pre-trained on the target texts. Recent studies focus on the generated texts and compute perplexities, which are superficial features and not reliable. In this study, we propose to utilize the probing technique for pre-training data detection by examining the model's internal activations. Our method is simple and effective and leads to more trustworthy pre-training data detection. Additionally, we propose ArxivMIA, a new challenging benchmark comprising arxiv abstracts from Computer Science and Mathematics categories. Our experiments demonstrate that our method outperforms all baselines, and achieves state-of-the-art performance on both WikiMIA and ArxivMIA, with additional experiments confirming its efficacy (Our code and dataset are available at https://github.com/zhliu0106/probing-lm-data).

cs.CL↗

arXiv2024

Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark

This paper presents a new tool learning dataset Seal-Tools, which contains self-instruct API-like tools. Seal-Tools not only offers a large number of tools, but also includes instances which demonstrate the practical application of tools. Seeking to generate data on a large scale while ensuring reliability, we propose a self-instruct method to generate tools and instances, allowing precise control over the process. Moreover, our Seal-Tools contains hard instances that call multiple tools to complete the job, among which some are nested tool callings. For precise and comprehensive evaluation, we use strict format control and design three metrics from different dimensions. Therefore, Seal-Tools can serve as a new benchmark to evaluate the tool-calling ability of LLMs. Finally, we evaluate several prevalent LLMs and our finetuned model on Seal-Tools. The results show that current systems are far from perfect. The code, data and experiment results are available at https://github.com/fairyshine/Seal-Tools .

cs.CL↗

arXiv2024

Real-fluid Transport Property Computations Based on the Boltzmann-weighted Full-dimensional Potential Model

The intermolecular potential plays crucial roles in real-fluid interactions away from the ideal-gas equilibrium, such as supercritical fluid, high-enthalpy fluid, plasma interactions, etc. We propose a Boltzmann-weighted Full-dimensional (BWF) potential model for real-fluid computations. It includes diverse intermolecular interactions so as to determine the potential well, molecular diameter, dipole moment, polarizability of species without introducing bath gases, allowing more accurate descriptions of potential surfaces with more potential parameters. The anisotropy and temperature dependence of potential parameters are also considered by applying the Boltzmann weighting on all orientations. Through the high-level Symmetry-Adapted Perturbation Theory calculations, full-dimensional potential energy surface datasets are obtained in 432 orientations for each species. Subsequently, the Boltzmann-weighted Full-dimensional potential parameters are derived by training the dataset exceeding 5*106 data, including nonpolar and polar molecules, radicals, long-chain molecules, and ions. These BWF transport properties calculated by the BWF potential have been compared against the Lennard-Jones transport properties as well as experimental viscosity, mass diffusivity, and thermal conductivity coefficients. It shows discrepancies of viscosity coefficients within 1% and 5% for nonpolar and polar molecules, respectively. Furthermore, this potential model is applied to study radicals, long-chain molecules, and ions, for which the experimental data is rarely accessed in high accuracy. It indicates significant prediction improvements of complex interactions between various particles. The new transport properties are also embedded to predict the laminar flame speeds and the flame extinction limits of methane, dimethyl ether, and n-heptane at elevated pressures, confirming its predictivity and effectiveness.

physics.app-ph↗

arXiv2024

MoPE: Mixture of Prefix Experts for Zero-Shot Dialogue State Tracking

Zero-shot dialogue state tracking (DST) transfers knowledge to unseen domains, reducing the cost of annotating new datasets. Previous zero-shot DST models mainly suffer from domain transferring and partial prediction problems. To address these challenges, we propose Mixture of Prefix Experts (MoPE) to establish connections between similar slots in different domains, which strengthens the model transfer performance in unseen domains. Empirical results demonstrate that MoPE-DST achieves the joint goal accuracy of 57.13% on MultiWOZ2.1 and 55.40% on SGD.

cs.CL↗

arXiv2024

Controllable and Diverse Data Augmentation with Large Language Model for Low-Resource Open-Domain Dialogue Generation

Data augmentation (DA) is crucial to mitigate model training instability and over-fitting problems in low-resource open-domain dialogue generation. However, traditional DA methods often neglect semantic data diversity, restricting the overall quality. Recently, large language models (LLM) have been used for DA to generate diversified dialogues. However, they have limited controllability and tend to generate dialogues with a distribution shift compared to the seed dialogues. To maximize the augmentation diversity and address the controllability problem, we propose \textbf{S}ummary-based \textbf{D}ialogue \textbf{A}ugmentation with LLM (SDA). Our approach enhances the controllability of LLM by using dialogue summaries as a planning tool. Based on summaries, SDA can generate high-quality and diverse dialogue data even with a small seed dataset. To evaluate the efficacy of data augmentation methods for open-domain dialogue, we designed a clustering-based metric to characterize the semantic diversity of the augmented dialogue data. The experimental results show that SDA can augment high-quality and semantically diverse dialogues given a small seed dataset and an LLM, and the augmented data can boost the performance of open-domain dialogue models.

cs.CL↗

arXiv2023

Mirror: A Universal Framework for Various Information Extraction Tasks

Sharing knowledge between information extraction tasks has always been a challenge due to the diverse data formats and task variations. Meanwhile, this divergence leads to information waste and increases difficulties in building complex applications in real scenarios. Recent studies often formulate IE tasks as a triplet extraction problem. However, such a paradigm does not support multi-span and n-ary extraction, leading to weak versatility. To this end, we reorganize IE problems into unified multi-slot tuples and propose a universal framework for various IE tasks, namely Mirror. Specifically, we recast existing IE tasks as a multi-span cyclic graph extraction problem and devise a non-autoregressive graph decoding algorithm to extract all spans in a single step. It is worth noting that this graph structure is incredibly versatile, and it supports not only complex IE tasks, but also machine reading comprehension and classification tasks. We manually construct a corpus containing 57 datasets for model pretraining, and conduct experiments on 30 datasets across 8 downstream tasks. The experimental results demonstrate that our model has decent compatibility and outperforms or reaches competitive performance with SOTA systems under few-shot and zero-shot settings. The code, model weights, and pretraining corpus are available at https://github.com/Spico197/Mirror .

cs.CL↗

arXiv2023

Thermal Magnetoelectrics in all Inorganic Quasi-Two-Dimensional Halide Perovskites

From lithium-ion batteries to high-temperature superconductors, oxide materials have been widely used in electronic devices. However, demands of future technologies require materials beyond oxides, as anion chemistries distinct from oxygen can expand the palette of mechanisms and phenomena, to achieve superior functionalities. Examples include nitride-based wide bandgap semiconductors and halide perovskite solar cells, with MAPbBr3 being a representation revolutionizing photovoltaics research. Here, we demonstrate magnetoelectric behaviour in quasi-two-dimensional halides (K,Rb)3Mn2Cl7 through simultaneous thermal control of electric and magnetic polarizations by exploiting a polar-to-antipolar displacive transition. Additionally, our calculations indicate a possible polarization switching path including a strong magnetoelectric coupling, indicating halides can be excellent platforms to design future multiferroic and ferroelectric devices. We expect our findings to broaden the exploration of multiferroics to non-oxide materials and open access to novel mechanisms, beyond conventional electric/magnetic control, for coupling ferroic orders.

cond-mat.mtrl-sci↗

arXiv2023

An Investigation of Darwiche and Pearl's Postulates for Iterated Belief Update

Belief revision and update, two significant types of belief change, both focus on how an agent modify her beliefs in presence of new information. The most striking difference between them is that the former studies the change of beliefs in a static world while the latter concentrates on a dynamically-changing world. The famous AGM and KM postulates were proposed to capture rational belief revision and update, respectively. However, both of them are too permissive to exclude some unreasonable changes in the iteration. In response to this weakness, the DP postulates and its extensions for iterated belief revision were presented. Furthermore, Rodrigues integrated these postulates in belief update. Unfortunately, his approach does not meet the basic requirement of iterated belief update. This paper is intended to solve this problem of Rodrigues's approach. Firstly, we present a modification of the original KM postulates based on belief states. Subsequently, we migrate several well-known postulates for iterated belief revision to iterated belief update. Moreover, we provide the exact semantic characterizations based on partial preorders for each of the proposed postulates. Finally, we analyze the compatibility between the above iterated postulates and the KM postulates for belief update.

cs.AI↗

arXiv2023

CED: Catalog Extraction from Documents

Sentence-by-sentence information extraction from long documents is an exhausting and error-prone task. As the indicator of document skeleton, catalogs naturally chunk documents into segments and provide informative cascade semantics, which can help to reduce the search space. Despite their usefulness, catalogs are hard to be extracted without the assist from external knowledge. For documents that adhere to a specific template, regular expressions are practical to extract catalogs. However, handcrafted heuristics are not applicable when processing documents from different sources with diverse formats. To address this problem, we build a large manually annotated corpus, which is the first dataset for the Catalog Extraction from Documents (CED) task. Based on this corpus, we propose a transition-based framework for parsing documents into catalog trees. The experimental results demonstrate that our proposed method outperforms baseline systems and shows a good ability to transfer. We believe the CED task could fill the gap between raw text segments and information extraction tasks on extremely long documents. Data and code are available at \url{https://github.com/Spico197/CatalogExtraction}

cs.CL↗

arXiv2023

Inorganic Tin Perovskites with Tunable Conductivity Enabled by Organic Modifiers

Achieving control over the transport properties of charge-carriers is a crucial aspect of realizing high-performance electronic materials. In metal-halide perovskites, which offer convenient manufacturing traits and tunability for certain optoelectronic applications, this is challenging: The perovskite structure itself, poses fundamental limits to maximum dopant incorporation. Here, we demonstrate an organic modifier incorporation strategy capable of modulating the electronic density of states in halide tin perovskites without altering the perovskite lattice, in a similar fashion to substitutional doping in traditional semiconductors. By incorporating organic small molecules and conjugated polymers into cesium tin iodide (CsSnI3) perovskites, we achieve carrier density tunability over 2.7 decades, transition from a semiconducting to a metallic nature, and high electrical conductivity exceeding 200 S/cm. We leverage these tunable and enhanced electronic properties to achieve a thin-film, lead free, thermoelectric material with a near room-temperature figure-of-merit (ZT) of 0.21, the highest amongst all halide perovskite thermoelectrics. Our strategy provides an additional degree of freedom in the design of halide perovskites for optoelectronic and energy applications.

physics.app-ph↗

arXiv2023

Closed-loop Error Correction Learning Accelerates Experimental Discovery of Thermoelectric Materials

The exploration of thermoelectric materials is challenging considering the large materials space, combined with added exponential degrees of freedom coming from doping and the diversity of synthetic pathways. Here we seek to incorporate historical data and update and refine it using experimental feedback by employing error-correction learning (ECL). We thus learn from prior datasets and then adapt the model to differences in synthesis and characterization that are otherwise difficult to parameterize. We then apply this strategy to discovering thermoelectric materials where we prioritize synthesis at temperatures < 300°C. We document a previously unreported chemical family of thermoelectric materials, PbSe:SnSb, finding that the best candidate in this chemical family, 2 wt% SnSb doped PbSe, exhibits a power factor more than 2x that of PbSe. Our investigations show that our closed-loop experimentation strategy reduces the required number of experiments to find an optimized material by as much as 3x compared to high-throughput searches powered by state-of-the-art machine learning models. We also observe that this improvement is dependent on the accuracy of prior in a manner that exhibits diminishing returns, and after a certain accuracy is reached, it is factors associated with experimental pathways that dictate the trends.

cond-mat.mtrl-sci↗

arXiv2023

Coupling photogeneration with thermodynamic modeling of light-induced alloy segregation enables the discovery of stabilizing dopants

We developed a generalized model that considers light as energy contributed through the thermalization of excited carriers to generate excitation-intensity- and temperature-dependent phase diagrams. We find that the model replicates the light-induced phase segregation behavior of the MAPb(I,Br)3 system. From there, we sought to study how best to design new, stable, mixed-halide alloys. The resultant first-principles predictions show that the pseudo halide anion BF4- suppresses phase segregation in FA0.83Cs0.17Pb(I0.6Br0.4)3, resulting in enhanced operating stability. The findings reveal that photostability is linked with the structure and electronic properties of materials and may be overcome using informed alloying strategies.

cond-mat.mtrl-sci↗