arXiv subjects

Qing Li

Publications and source records attributed to Qing Li.

At least 361 records · Page 20Linked to original sources

arXiv2024

Automated discovery of symbolic laws governing skill acquisition from naturally occurring data

Skill acquisition is a key area of research in cognitive psychology as it encompasses multiple psychological processes. The laws discovered under experimental paradigms are controversial and lack generalizability. This paper aims to unearth the laws of skill learning from large-scale training log data. A two-stage algorithm was developed to tackle the issues of unobservable cognitive states and algorithmic explosion in searching. Initially a deep learning model is employed to determine the learner's cognitive state and assess the feature importance. Subsequently, symbolic regression algorithms are utilized to parse the neural network model into algebraic equations. Experimental results show the algorithm can accurately restore preset laws within a noise range in continuous feedback settings. When applied to Lumosity training data, the method outperforms traditional and recent models in fitness terms. The study reveals two new forms of skill acquisition laws and reaffirms some previous findings.

cs.LG↗

arXiv2024

Asymmetric Drift Map of the Milky Way disk Populations between 8$-$16 kpc with LAMOST and Gaia datasets

The application of asymmetric drift (AD) tomography across different populations provides valuable insights into the kinematics, dynamics, and rotation curves of the Galactic disk. By leveraging common stars identified in both the LAMOST and Gaia surveys, alongside Gaia DR3's circular velocity curve, we conducted a qualitative exploration of asymmetric drift distributions within the Galactic disk spanning distances from 8 to 16 kpc. In the R-Z plane, we observed that the asymmetric drift is minimal near the mid-plane of the Galactic disk and gradually increases with vertical distance, resulting in a distinctive ``horn" shape. Additionally, our analysis revealed that populations with higher [$α$/Fe] ratios exhibit greater asymmetric drift compared to those with lower [$α$/Fe] ratios. Specifically, we found the asymmetric drift around the solar location to be approximately 6 km s$^{-1}$, with a median value of 16 km s$^{-1}$ across the entire sample. Notably, the median asymmetric drift in the northern region of the Galactic disk (20 km s$^{-1}$) surpasses that in the southern region (13 km s$^{-1}$), with errors remaining within 2 km s$^{-1}$. Furthermore, our investigation into mono-age stellar populations unveiled that older stellar populations tend to exhibit larger asymmetric drift and velocity dispersion, aligning closely with predictions from previous numerical models. Finally, based on chemical compositions, we observed that the median asymmetric drift of the thick disk significantly exceeds that of the thin disk and found that star formation within the thick disk primarily occurred earlier than 8-10 billion years, whereas the thin disk's predominant star formation period spanned 6-8 billion years ago.

astro-ph.GA↗

arXiv2024

MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets

Driven by powerful image diffusion models, recent research has achieved the automatic creation of 3D objects from textual or visual guidance. By performing score distillation sampling (SDS) iteratively across different views, these methods succeed in lifting 2D generative prior to the 3D space. However, such a 2D generative image prior bakes the effect of illumination and shadow into the texture. As a result, material maps optimized by SDS inevitably involve spurious correlated components. The absence of precise material definition makes it infeasible to relight the generated assets reasonably in novel scenes, which limits their application in downstream scenarios. In contrast, humans can effortlessly circumvent this ambiguity by deducing the material of the object from its appearance and semantics. Motivated by this insight, we propose MaterialSeg3D, a 3D asset material generation framework to infer underlying material from the 2D semantic prior. Based on such a prior model, we devise a mechanism to parse material in 3D space. We maintain a UV stack, each map of which is unprojected from a specific viewpoint. After traversing all viewpoints, we fuse the stack through a weighted voting scheme and then employ region unification to ensure the coherence of the object parts. To fuel the learning of semantics prior, we collect a material dataset, named Materialized Individual Objects (MIO), which features abundant images, diverse categories, and accurate annotations. Extensive quantitative and qualitative experiments demonstrate the effectiveness of our method.

cs.CV↗

arXiv2024

Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features

This paper investigates the dynamics of a deep neural network (DNN) learning interactions. Previous studies have discovered and mathematically proven that given each input sample, a well-trained DNN usually only encodes a small number of interactions (non-linear relationships) between input variables in the sample. A series of theorems have been derived to prove that we can consider the DNN's inference equivalent to using these interactions as primitive patterns for inference. In this paper, we discover the DNN learns interactions in two phases. The first phase mainly penalizes interactions of medium and high orders, and the second phase mainly learns interactions of gradually increasing orders. We can consider the two-phase phenomenon as the starting point of a DNN learning over-fitted features. Such a phenomenon has been widely shared by DNNs with various architectures trained for different tasks. Therefore, the discovery of the two-phase dynamics provides a detailed mechanism for how a DNN gradually learns different inference patterns (interactions). In particular, we have also verified the claim that high-order interactions have weaker generalization power than low-order interactions. Thus, the discovered two-phase dynamics also explains how the generalization power of a DNN changes during the training process.

cs.LG↗

arXiv2024

An Embodied Generalist Agent in 3D World

Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e.g., 3D grounding, embodied reasoning and acting. We argue these limitations significantly hinder current models from performing real-world tasks and approaching general intelligence. To this end, we introduce LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. LEO is trained with a unified task interface, model architecture, and objective in two stages: (i) 3D vision-language (VL) alignment and (ii) 3D vision-language-action (VLA) instruction tuning. We collect large-scale datasets comprising diverse object-level and scene-level tasks, which require considerable understanding of and interaction with the 3D world. Moreover, we meticulously design an LLM-assisted pipeline to produce high-quality 3D VL data. Through extensive experiments, we demonstrate LEO's remarkable proficiency across a wide spectrum of tasks, including 3D captioning, question answering, embodied reasoning, navigation and manipulation. Our ablative studies and scaling analyses further provide valuable insights for developing future embodied generalist agents. Code and data are available on project page.

cs.CV↗

arXiv2024

Room temperature Si:S barrier infrared detector with broadband response up to 4.4μm

Mid-infrared spectrum is a critical tool for chemical analysis, industrial inspection, environment, and other fields due to its rich chemical bond information. However, the complicated growth or fabrication procedures of existing mid-infrared sensitive materials hinder the large-scale production and utilization of mid-infrared detectors. To address this issue, we developed Si:S barrier detectors employing sulfur doped silicon and a sophisticated band barrier design. Since the transport of dark current and photo current is separated, the barrier design effectively suppresses the dark current while allowing the photo current to leverage gain mechanisms, thereby substantially improving signal-to-noise ratio. As a result, the detector exhibits an infrared response range covering from 1.12 to 4.4μm with a peak at 3.3μm, excluding its intrinsic response in visible range. Its peak quantum efficiency surpasses that of the best mid-infrared silicon-based detector reported to date by an order of magnitude, reaching 2% at room temperature. The peak detectivity at 90K is 1.4E11 Jones @1.4V and decreases to 4.4E9 Jones @1.4V, 210K, comparable to the typical III-V and IV-VI photodetectors at one thousandth fabrication cost. Leveraging the well-established silicon-based manufacturing process, this device holds promise for large-scale production at a reduced price, offering a cost-effective solution for future mid-infrared detection.

physics.app-ph↗

arXiv2024

Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation

Large Language Models (LLMs) have made significant strides in information acquisition. However, their overreliance on potentially flawed parametric knowledge leads to hallucinations and inaccuracies, particularly when handling long-tail, domain-specific queries. Retrieval Augmented Generation (RAG) addresses this limitation by incorporating external, non-parametric knowledge. Nevertheless, the retrieved long-context documents often contain noisy, irrelevant information alongside vital knowledge, negatively diluting LLMs' attention. Inspired by the supportive role of essential concepts in individuals' reading comprehension, we propose a novel concept-based RAG framework with the Abstract Meaning Representation (AMR)-based concept distillation algorithm. The proposed algorithm compresses the cluttered raw retrieved documents into a compact set of crucial concepts distilled from the informative nodes of AMR by referring to reliable linguistic features. The concepts explicitly constrain LLMs to focus solely on vital information in the inference process. We conduct extensive experiments on open-domain question-answering datasets to empirically evaluate the proposed method's effectiveness. The results indicate that the concept-based RAG framework outperforms other baseline methods, particularly as the number of supporting documents increases, while also exhibiting robustness across various backbone LLMs. This emphasizes the distilled concepts are informative for augmenting the RAG process by filtering out interference information. To the best of our knowledge, this is the first work introducing AMR to enhance the RAG, presenting a potential solution to augment inference performance with semantic-based context compression.

cs.CL↗

arXiv2024

SlotGAT: Slot-based Message Passing for Heterogeneous Graph Neural Network

Heterogeneous graphs are ubiquitous to model complex data. There are urgent needs on powerful heterogeneous graph neural networks to effectively support important applications. We identify a potential semantic mixing issue in existing message passing processes, where the representations of the neighbors of a node $v$ are forced to be transformed to the feature space of $v$ for aggregation, though the neighbors are in different types. That is, the semantics in different node types are entangled together into node $v$'s representation. To address the issue, we propose SlotGAT with separate message passing processes in slots, one for each node type, to maintain the representations in their own node-type feature spaces. Moreover, in a slot-based message passing layer, we design an attention mechanism for effective slot-wise message aggregation. Further, we develop a slot attention technique after the last layer of SlotGAT, to learn the importance of different slots in downstream tasks. Our analysis indicates that the slots in SlotGAT can preserve different semantics in various feature spaces. The superiority of SlotGAT is evaluated against 13 baselines on 6 datasets for node classification and link prediction. Our code is at https://github.com/scottjiao/SlotGAT_ICML23/.

cs.LG↗

arXiv2024

Neural-Symbolic Recursive Machine for Systematic Generalization

Current learning models often struggle with human-like systematic generalization, particularly in learning compositional rules from limited data and extrapolating them to novel combinations. We introduce the Neural-Symbolic Recursive Machine (NSR), whose core is a Grounded Symbol System (GSS), allowing for the emergence of combinatorial syntax and semantics directly from training data. The NSR employs a modular design that integrates neural perception, syntactic parsing, and semantic reasoning. These components are synergistically trained through a novel deduction-abduction algorithm. Our findings demonstrate that NSR's design, imbued with the inductive biases of equivariance and compositionality, grants it the expressiveness to adeptly handle diverse sequence-to-sequence tasks and achieve unparalleled systematic generalization. We evaluate NSR's efficacy across four challenging benchmarks designed to probe systematic generalization capabilities: SCAN for semantic parsing, PCFG for string manipulation, HINT for arithmetic reasoning, and a compositional machine translation task. The results affirm NSR's superiority over contemporary neural and hybrid models in terms of generalization and transferability.

cs.LG↗

arXiv2024

Recommender Systems in the Era of Large Language Models (LLMs)

With the prosperity of e-commerce and web applications, Recommender Systems (RecSys) have become an important component of our daily life, providing personalized suggestions that cater to user preferences. While Deep Neural Networks (DNNs) have made significant advancements in enhancing recommender systems by modeling user-item interactions and incorporating textual side information, DNN-based methods still face limitations, such as difficulties in understanding users' interests and capturing textual side information, inabilities in generalizing to various recommendation scenarios and reasoning on their predictions, etc. Meanwhile, the emergence of Large Language Models (LLMs), such as ChatGPT and GPT4, has revolutionized the fields of Natural Language Processing (NLP) and Artificial Intelligence (AI), due to their remarkable abilities in fundamental responsibilities of language understanding and generation, as well as impressive generalization and reasoning capabilities. As a result, recent studies have attempted to harness the power of LLMs to enhance recommender systems. Given the rapid evolution of this research direction in recommender systems, there is a pressing need for a systematic overview that summarizes existing LLM-empowered recommender systems, to provide researchers in relevant fields with an in-depth understanding. Therefore, in this paper, we conduct a comprehensive review of LLM-empowered recommender systems from various aspects including Pre-training, Fine-tuning, and Prompting. More specifically, we first introduce representative methods to harness the power of LLMs (as a feature encoder) for learning representations of users and items. Then, we review recent techniques of LLMs for enhancing recommender systems from three paradigms, namely pre-training, fine-tuning, and prompting. Finally, we comprehensively discuss future directions in this emerging field.

cs.IR↗

arXiv2024

Variational Tracking and Redetection for Closely-spaced Objects in Heavy Clutter: Supplementary Materials

The non-homogeneous Poisson process (NHPP) is a widely used measurement model that allows for an object to generate multiple measurements over time. However, it can be difficult to efficiently and reliably track multiple objects under this NHPP model in scenarios with a high density of closely-spaced objects and heavy clutter. Therefore, based on the general coordinate ascent variational filtering framework, this paper presents a variational Bayes association-based NHPP tracker (VB-AbNHPP) that can efficiently perform tracking, data association, and learning of target and clutter rates with a parallelisable implementation. In addition, a variational localisation strategy is proposed, which enables rapid rediscovery of missed targets from a large surveillance area under extremely heavy clutter. This strategy is integrated into the VB-AbNHPP tracker, resulting in a robust methodology that can automatically detect and recover from track loss. This tracker demonstrates improved tracking performance compared with existing trackers in challenging scenarios, in terms of both accuracy and efficiency.

eess.SP↗

arXiv2024

Physical Vapor Deposition of High Mobility P-type Tellurium and its Applications for Gate-tunable van der Waals PN Photodiodes

Recently tellurium (Te) has attracted resurgent interests due to its p-type characteristics and outstanding ambient environmental stability. Here we present a substrate engineering based physical vapor deposition method to synthesize high-quality Te nanoflakes and achieved a field-effect hole mobility of 1500 cm2/Vs, which is, to the best of our knowledge, the highest among the existing synthesized van der Waals p-type semiconductors. The high mobility Te enables the fabrication of Te/MoS2 pn diodes with highly gate-tunable electronic and optoelectronic characteristics. The Te/MoS2 heterostructure can be used as a visible range photodetector with a current responsivity up to 630 A/W, which is about one order of magnitude higher than the one achieved using p-type Si-MoS2 PN photodiodes. The photo response of the Te/MoS2 heterojunction also exhibits strong gate tunability due to their ultrathin thickness and unique band structures. The successful synthesis of high mobility Te and the enabled Te/MoS2 photodiodes show promise for the development of highly tunable and ultrathin photodetectors.

physics.app-ph↗

arXiv2024

Record high superconducting transition temperature in Ti$_{1-x}$Mn$_x$ alloy with rich magnetic element Mn

It is well-known that magnetic moments are very harmful to superconductivity. A typical example is the element Mn whose compounds usually exhibit strong magnetism. Thus, it is very hard to achieve superconductivity in materials containing Mn. Here, we report enhanced superconductivity with the superconducting transition temperature ($T_\text{c}$) up to a record high-value of about 26 K in a beta-phase Ti$_{1-x}$Mn$_x$ alloy containing rich magnetic element Mn under high pressures. This is contrary to the intuition that the magnetic moments always suppress superconductivity. Under high pressures, we also found that in the middle-pressure regime, the Pauli limit of the upper critical field is surpassed. The synchrotron X-ray diffraction data shows an unchanged beta-phase with a continuous contraction of the cell volume, which is well supported by the first-principles calculations. Although the theoretical results based on electron-phonon coupling (EPC) can interpret the $T_\text{c}$ value in a certain pressure region, the monotonic enhancement of superconductivity by pressure cannot seek support from the theory. Our results show a surprising enhancement of superconductivity in Ti$_{1-x}$Mn$_x$ alloy with a considerable Mn content.

cond-mat.supr-con↗

arXiv2024

Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective

Molecule discovery plays a crucial role in various scientific fields, advancing the design of tailored materials and drugs. However, most of the existing methods heavily rely on domain experts, require excessive computational cost, or suffer from sub-optimal performance. On the other hand, Large Language Models (LLMs), like ChatGPT, have shown remarkable performance in various cross-modal tasks due to their powerful capabilities in natural language understanding, generalization, and in-context learning (ICL), which provides unprecedented opportunities to advance molecule discovery. Despite several previous works trying to apply LLMs in this task, the lack of domain-specific corpus and difficulties in training specialized LLMs still remain challenges. In this work, we propose a novel LLM-based framework (MolReGPT) for molecule-caption translation, where an In-Context Few-Shot Molecule Learning paradigm is introduced to empower molecule discovery with LLMs like ChatGPT to perform their in-context learning capability without domain-specific pre-training and fine-tuning. MolReGPT leverages the principle of molecular similarity to retrieve similar molecules and their text descriptions from a local database to enable LLMs to learn the task knowledge from context examples. We evaluate the effectiveness of MolReGPT on molecule-caption translation, including molecule understanding and text-based molecule generation. Experimental results show that compared to fine-tuned models, MolReGPT outperforms MolT5-base and is comparable to MolT5-large without additional training. To the best of our knowledge, MolReGPT is the first work to leverage LLMs via in-context learning in molecule-caption translation for advancing molecule discovery. Our work expands the scope of LLM applications, as well as providing a new paradigm for molecule discovery and design.

cs.CL↗

arXiv2024

DeviceRadar: Online IoT Device Fingerprinting in ISPs using Programmable Switches

Device fingerprinting can be used by Internet Service Providers (ISPs) to identify vulnerable IoT devices for early prevention of threats. However, due to the wide deployment of middleboxes in ISP networks, some important data, e.g., 5-tuples and flow statistics, are often obscured, rendering many existing approaches invalid. It is further challenged by the high-speed traffic of hundreds of terabytes per day in ISP networks. This paper proposes DeviceRadar, an online IoT device fingerprinting framework that achieves accurate, real-time processing in ISPs using programmable switches. We innovatively exploit "key packets" as a basis of fingerprints only using packet sizes and directions, which appear periodically while exhibiting differences across different IoT devices. To utilize them, we propose a packet size embedding model to discover the spatial relationships between packets. Meanwhile, we design an algorithm to extract the "key packets" of each device, and propose an approach that jointly considers the spatial relationships and the key packets to produce a neighboring key packet distribution, which can serve as a feature vector for machine learning models for inference. Last, we design a model transformation method and a feature extraction process to deploy the model on a programmable data plane within its constrained arithmetic operations and memory to achieve line-speed processing. Our experiments show that DeviceRadar can achieve state-of-the-art accuracy across 77 IoT devices with 40 Gbps throughput, and requires only 1.3% of the processing time compared to GPU-accelerated approaches.

cs.NI↗

arXiv2024

Advancing the Robustness of Large Language Models through Self-Denoised Smoothing

Although large language models (LLMs) have achieved significant success, their vulnerability to adversarial perturbations, including recent jailbreak attacks, has raised considerable concerns. However, the increasing size of these models and their limited access make improving their robustness a challenging task. Among various defense strategies, randomized smoothing has shown great potential for LLMs, as it does not require full access to the model's parameters or fine-tuning via adversarial training. However, randomized smoothing involves adding noise to the input before model prediction, and the final model's robustness largely depends on the model's performance on these noise corrupted data. Its effectiveness is often limited by the model's sub-optimal performance on noisy data. To address this issue, we propose to leverage the multitasking nature of LLMs to first denoise the noisy inputs and then to make predictions based on these denoised versions. We call this procedure self-denoised smoothing. Unlike previous denoised smoothing techniques in computer vision, which require training a separate model to enhance the robustness of LLMs, our method offers significantly better efficiency and flexibility. Our experimental results indicate that our method surpasses existing methods in both empirical and certified robustness in defending against adversarial attacks for both downstream tasks and human alignments (i.e., jailbreak attacks). Our code is publicly available at https://github.com/UCSB-NLP-Chang/SelfDenoise

cs.CL↗

arXiv2024

Strong-coupling superconductivity and weak vortex pinning in Ta-doped CsV$_{3}$Sb$_{5}$ single crystals

By measuring magnetizations of pristine and Ta-doped CsV$_{3}$Sb$_{5}$ single crystals, we have carried out systematic studies on the lower critical field, critical current density, and equilibrium magnetization of this kagome system. The lower critical field has been investigated in the two typical samples, and the temperature dependent lower critical field obtained in Ta-doped sample can be fitted by using the model with two $s$-wave superconducting gaps yielding the larger gap of $2Δ_{s1}/k_\mathrm{B}T_\mathrm{c}=7.9\;(\pm1.8)$. This indicates a strong-coupling feature of the V-based superconductors. The measured magnetization hysteresis loops allow us to calculate the critical current density, which shows a very weak bulk vortex pinning. The magnetization hysteresis loops measured in these two kinds of samples can be well described by a recently proposed generalized phenomenological model, which leads to the determination of many fundamental parameters for these superconductors. Our systematic results and detailed analysis conclude that this V-based kagome system has features of strong-coupling superconductivity, relatively large Ginzburg-Landau parameter and weak vortex coupling.

cond-mat.supr-con↗

arXiv2024

Local pairing versus bulk superconductivity intertwined by the charge density wave order in Cs(V$_{1-x}$Ta$_{x}$)$_{3}$Sb$_{5}$

There is a common belief that superconductivity and charge density wave (CDW) order accommodate homogenously in real space but compete with each other for the effective density of states in momentum space in CDW superconductors. By measuring resistivity along the $c$-axis in Cs(V$_{1-x}$Ta$_{x}$)$_{3}$Sb$_{5}$, we observe strong superconducting fluctuation behavior coexisting with the CDW order in the pristine CsV$_{3}$Sb$_{5}$, and the fluctuation region becomes narrowed when the Ta doping suppresses the CDW order. The onset transition temperature barely changes with the Ta doping. Therefore, the bulk superconductivity may be established by a doping-independent local pairing, and it can be suppressed in some regions by the spatially variable CDW order along the $c$-axis. Our results violate the above-mentioned belief about CDW superconductors and demonstrate the intricate interaction between superconductivity and CDW order in this kagome superconductor.

cond-mat.supr-con↗