arXiv subjects

Zhe Li

Publications and source records attributed to Zhe Li.

At least 163 records · Page 9Linked to original sources

arXiv2025

Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs

In this technical report, we tackle the challenges of training large-scale Mixture of Experts (MoE) models, focusing on overcoming cost inefficiency and resource limitations prevalent in such systems. To address these issues, we present two differently sized MoE large language models (LLMs), namely Ling-Lite and Ling-Plus (referred to as "Bailing" in Chinese, spelled Bǎilíng in Pinyin). Ling-Lite contains 16.8 billion parameters with 2.75 billion activated parameters, while Ling-Plus boasts 290 billion parameters with 28.8 billion activated parameters. Both models exhibit comparable performance to leading industry benchmarks. This report offers actionable insights to improve the efficiency and accessibility of AI development in resource-constrained settings, promoting more scalable and sustainable technologies. Specifically, to reduce training costs for large-scale MoE models, we propose innovative methods for (1) optimization of model architecture and training processes, (2) refinement of training anomaly handling, and (3) enhancement of model evaluation efficiency. Additionally, leveraging high-quality data generated from knowledge graphs, our models demonstrate superior capabilities in tool use compared to other models. Ultimately, our experimental findings demonstrate that a 300B MoE LLM can be effectively trained on lower-performance devices while achieving comparable performance to models of a similar scale, including dense and MoE models. Compared to high-performance devices, utilizing a lower-specification hardware system during the pre-training phase demonstrates significant cost savings, reducing computing costs by approximately 20%. The models can be accessed at https://huggingface.co/inclusionAI.

cs.LG↗

arXiv2025

Gemini Embedding: Generalizable Embeddings from Gemini

In this report, we introduce Gemini Embedding, a state-of-the-art embedding model leveraging the power of Gemini, Google's most capable large language model. Capitalizing on Gemini's inherent multilingual and code understanding capabilities, Gemini Embedding produces highly generalizable embeddings for text spanning numerous languages and textual modalities. The representations generated by Gemini Embedding can be precomputed and applied to a variety of downstream tasks including classification, similarity, clustering, ranking, and retrieval. Evaluated on the Massive Multilingual Text Embedding Benchmark (MMTEB), which includes over one hundred tasks across 250+ languages, Gemini Embedding substantially outperforms prior state-of-the-art models, demonstrating considerable improvements in embedding quality. Achieving state-of-the-art performance across MMTEB's multilingual, English, and code benchmarks, our unified model demonstrates strong capabilities across a broad selection of tasks and surpasses specialized domain-specific models.

cs.CL↗

arXiv2025

LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioning

Language plays a vital role in the realm of human motion. Existing methods have largely depended on CLIP text embeddings for motion generation, yet they fall short in effectively aligning language and motion due to CLIP's pretraining on static image-text pairs. This work introduces LaMP, a novel Language-Motion Pretraining model, which transitions from a language-vision to a more suitable language-motion latent space. It addresses key limitations by generating motion-informative text embeddings, significantly enhancing the relevance and semantics of generated motion sequences. With LaMP, we advance three key tasks: text-to-motion generation, motion-text retrieval, and motion captioning through aligned language-motion representation learning. For generation, we utilize LaMP to provide the text condition instead of CLIP, and an autoregressive masked prediction is designed to achieve mask modeling without rank collapse in transformers. For retrieval, motion features from LaMP's motion transformer interact with query tokens to retrieve text features from the text transformer, and vice versa. For captioning, we finetune a large language model with the language-informative motion features to develop a strong motion captioning model. In addition, we introduce the LaMP-BertScore metric to assess the alignment of generated motions with textual descriptions. Extensive experimental results on multiple datasets demonstrate substantial improvements over previous methods across all three tasks. The code of our method will be made public.

cs.CV↗

arXiv2025

GaussianCAD: Robust Self-Supervised CAD Reconstruction from Three Orthographic Views Using 3D Gaussian Splatting

The automatic reconstruction of 3D computer-aided design (CAD) models from CAD sketches has recently gained significant attention in the computer vision community. Most existing methods, however, rely on vector CAD sketches and 3D ground truth for supervision, which are often difficult to be obtained in industrial applications and are sensitive to noise inputs. We propose viewing CAD reconstruction as a specific instance of sparse-view 3D reconstruction to overcome these limitations. While this reformulation offers a promising perspective, existing 3D reconstruction methods typically require natural images and corresponding camera poses as inputs, which introduces two major significant challenges: (1) modality discrepancy between CAD sketches and natural images, and (2) difficulty of accurate camera pose estimation for CAD sketches. To solve these issues, we first transform the CAD sketches into representations resembling natural images and extract corresponding masks. Next, we manually calculate the camera poses for the orthographic views to ensure accurate alignment within the 3D coordinate system. Finally, we employ a customized sparse-view 3D reconstruction method to achieve high-quality reconstructions from aligned orthographic views. By leveraging raster CAD sketches for self-supervision, our approach eliminates the reliance on vector CAD sketches and 3D ground truth. Experiments on the Sub-Fusion360 dataset demonstrate that our proposed method significantly outperforms previous approaches in CAD reconstruction performance and exhibits strong robustness to noisy inputs.

cs.CV↗

arXiv2025

Coexistence of topological surface states and superconductivity in Dirac semimetal NiTe$_2$

The coexistence of topological bands around the Fermi level ($E_F$) and superconductivity provides a fundamental platform for exploring their interplay. However, few materials inherently display both properties. In this study, we demonstrate the coexistence of topological surface states at the $E_F$ and superconductivity in NiTe$_2$ single crystals, a material hitherto not recognized as superconducting. Quasiparticle interference measurements performed via scanning tunneling microscopy suggest the presence of topological surface states at the $E_F$, which is further corroborated by density functional theory simulations. Experimental evidence for superconductivity is provided via electronic transport measurements and specific heat capacity analyses. Our results suggest that NiTe$_2$ represents a promising platform for investigating the rich interplay between topological states and superconductivity.

cond-mat.supr-con↗

arXiv2025

BodyGen: Advancing Towards Efficient Embodiment Co-Design

Embodiment co-design aims to optimize a robot's morphology and control policy simultaneously. While prior work has demonstrated its potential for generating environment-adaptive robots, this field still faces persistent challenges in optimization efficiency due to the (i) combinatorial nature of morphological search spaces and (ii) intricate dependencies between morphology and control. We prove that the ineffective morphology representation and unbalanced reward signals between the design and control stages are key obstacles to efficiency. To advance towards efficient embodiment co-design, we propose BodyGen, which utilizes (1) topology-aware self-attention for both design and control, enabling efficient morphology representation with lightweight model sizes; (2) a temporal credit assignment mechanism that ensures balanced reward signals for optimization. With our findings, Body achieves an average 60.03% performance improvement against state-of-the-art baselines. We provide codes and more results on the website: https://genesisorigin.github.io.

cs.RO↗

arXiv2025

Physics-Aware Inverse Design for Nanowire Single-Photon Avalanche Detectors via Deep Learning

Single-photon avalanche detectors (SPADs) have enabled various applications in emerging photonic quantum information technologies in recent years. However, despite many efforts to improve SPAD's performance, the design of SPADs remained largely an iterative and time-consuming process where a designer makes educated guesses of a device structure based on empirical reasoning and solves the semiconductor drift-diffusion model for it. In contrast, the inverse problem, i.e., directly inferring a structure needed to achieve desired performance, which is of ultimate interest to designers, remains an unsolved problem. We propose a novel physics-aware inverse design workflow for SPADs using a deep learning model and demonstrate it with an example of finding the key parameters of semiconductor nanowires constituting the unit cell of an SPAD, given target photon detection efficiency. Our inverse design workflow is not restricted to the case demonstrated and can be applied to design conventional planar structure-based SPADs, photodetectors, and solar cells.

physics.app-ph↗

arXiv2025

Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs

Large Vision-Language Models (LVLMs) have made significant strides in multimodal comprehension, thanks to extensive pre-training and fine-tuning on large-scale visual datasets. However, despite their robust textual safety mechanisms, they remain vulnerable to harmful visual inputs. Existing safeguards-typically relying on pre-filtering or fine-tuning-incur high costs and diminish overall utility. To address this critical vulnerability, we introduce SafeCLIP, a lightweight method that leverages LVLMs inherent multimodal alignment for zero-shot toxic image detection. By projecting CLIPs discarded CLS token into its text space and matching it with toxic descriptors, SafeCLIP detects harmful content without any architectural changes-adding minimal latency and enabling dynamic safety corrections during inference and fine-tuning.Experiments show that SafeCLIP achieves a 66.9% defense success rate with only 3.2% false positive rate and 7.2% overhead. In contrast, state-of-the-art methods achieve 52.9% success but have a 10.7% false positive rate and 210% overhead. Our work demonstrates that leveraging inherent multimodal alignment can yield efficient, low-cost LVLM safety. Code is available at anonymous.4open.science/r/safeclip-2C01.

cs.CL↗

arXiv2025

Semantically Robust Unsupervised Image Translation for Paired Remote Sensing Images

Image translation for change detection or classification in bi-temporal remote sensing images is unique. Although it can acquire paired images, it is still unsupervised. Moreover, strict semantic preservation in translation is always needed instead of multimodal outputs. In response to these problems, this paper proposes a new method, SRUIT (Semantically Robust Unsupervised Image-to-image Translation), which ensures semantically robust translation and produces deterministic output. Inspired by previous works, the method explores the underlying characteristics of bi-temporal Remote Sensing images and designs the corresponding networks. Firstly, we assume that bi-temporal Remote Sensing images share the same latent space, for they are always acquired from the same land location. So SRUIT makes the generators share their high-level layers, and this constraint will compel two domain mapping to fall into the same latent space. Secondly, considering land covers of bi-temporal images could evolve into each other, SRUIT exploits the cross-cycle-consistent adversarial networks to translate from one to the other and recover them. Experimental results show that constraints of sharing weights and cross-cycle consistency enable translated images with both good perceptual image quality and semantic preservation for significant differences.

cs.CV↗

arXiv2025

Spectral-Aware Low-Rank Adaptation for Speaker Verification

Previous research has shown that the principal singular vectors of a pre-trained model's weight matrices capture critical knowledge. In contrast, those associated with small singular values may contain noise or less reliable information. As a result, the LoRA-based parameter-efficient fine-tuning (PEFT) approach, which does not constrain the use of the spectral space, may not be effective for tasks that demand high representation capacity. In this study, we enhance existing PEFT techniques by incorporating the spectral information of pre-trained weight matrices into the fine-tuning process. We investigate spectral adaptation strategies with a particular focus on the additive adjustment of top singular vectors. This is accomplished by applying singular value decomposition (SVD) to the pre-trained weight matrices and restricting the fine-tuning within the top spectral space. Extensive speaker verification experiments on VoxCeleb1 and CN-Celeb1 demonstrate enhanced tuning performance with the proposed approach. Code is released at https://github.com/lizhepolyu/SpectralFT.

eess.AS↗

arXiv2025

Universally optimizable strategy for magnetic gaps towards high-temperature quantum anomalous Hall states via magnetic-insulator/topological-insulator building-blocks

Optimizing the magnetic Zeeman-splitting term, specifically the magnetic gap of the topological surface states (TSSs), is a crucial issue and central challenge in advancing higher-temperature quantum anomalous Hall (QAH) states. In this work, we demonstrate a counterintuitive, nonmonotonic relationship between the magnetic gap and the hybridization strength in ferromagnetic-insulator (FMI)/topological-insulator (TI) sandwich structures. Concretely, insufficient hybridization strength fails to induce a substantial magnetic gap; while excessive hybridization incandesces the competition between kinetic and Coulomb exchange interactions, thereby reducing the gap. Strong hybridization strength also spatially delocalizes the TSSs, diminishing the effective orbital coupling between TSS-based p and magnetic d orbitals, which further weakens kinetic and Coulomb exchange interaction strength. Moreover, modifying the stacking order offers an experimentally viable approach to optimizing magnetic gaps, enabling the tunability of Chern numbers, chirality and maximizing global gaps. These findings unveil a universal guiding principle for optimizing magnetic gaps in FMI-TI proximity-based QAH systems, offering valuable insights for advancing experimental implementations in this field.

cond-mat.mtrl-sci↗

arXiv2025

Demonstration of Time-reversal Symmetric Two-Dimensional Photonic Topological Anderson Insulator

Recently, the impact of disorder on topological properties has attracted significant attention in photonics, especially the intriguing disorder-induced topological phase transitions in photonic topological Anderson insulators (PTAIs). However, the reported PTAIs are based on time-reversal symmetry broken systems or quasi-three-dimensional time-reversal invariant system, both of which would limit the applications in integrated optics. Here, we realize a time-reversal symmetric two-dimensional PTAI on silicon platform within the near-IR wavelength range, taking the advantageous valley degree of freedom of photonic crystal. A low-threshold topological Anderson phase transition is observed by applying disorder to the critical topologically trivial phase. Conversely, we have also realized extremely robust topologically protected edge states based on the stable topological phase. Both two phenomena are validated through theoretical Dirac Hamiltonian analysis, numerical simulations, and experimental measurements. Our proposed structure holds promise to achieve near-zero topological phase transition thresholds, which breaks the conventional cognition that strong disorder is required to induce the phase transition. It significantly alleviates the difficulty of manipulating disorder and could be extended to other systems, such as condensed matter systems where strong disorder is hard to implement. This work is also beneficial to construct highly robust photonic integrated circuits serving for on-chip photonic and quantum optic information processing. Moreover, this work also provides an outstanding platform to investigate on-chip integrated disordered systems.

physics.optics↗

arXiv2025

Subtractive Training for Music Stem Insertion using Latent Diffusion Models

We present Subtractive Training, a simple and novel method for synthesizing individual musical instrument stems given other instruments as context. This method pairs a dataset of complete music mixes with 1) a variant of the dataset lacking a specific stem, and 2) LLM-generated instructions describing how the missing stem should be reintroduced. We then fine-tune a pretrained text-to-audio diffusion model to generate the missing instrument stem, guided by both the existing stems and the text instruction. Our results demonstrate Subtractive Training's efficacy in creating authentic drum stems that seamlessly blend with the existing tracks. We also show that we can use the text instruction to control the generation of the inserted stem in terms of rhythm, dynamics, and genre, allowing us to modify the style of a single instrument in a full song while keeping the remaining instruments the same. Lastly, we extend this technique to MIDI formats, successfully generating compatible bass, drum, and guitar parts for incomplete arrangements.

cs.SD↗

arXiv2025

Designing Guidance for Multiple Valley-based Topological States Driven by Magnetic Substrates: Potential Applications at High Temperatures

Valley-based topological phases offer a wealth of exotic quantum states with tunable functionalities, driven by the valley degree of freedom. In this work, by constructing heterostructures of germanene (silicene, stanene) on various magnetic substrates, we address key tuning factors such as the spin-orbit coupling (SOC) strength of the substrate, magnetic orientations, and stacking orders, all of which govern multiple valley-based topological features. We present a comprehensive guiding principle for the efficient manipulation of these features, achieved simply by designing and modulating the magnetic properties of the underlying substrates. Specifically, increasing the SOC strength of the magnetic substrate acilitates a range of topological phase transitions characterized by different Chern numbers, with many systems exhibiting a transition from quantum valley Hall to quantum anomalous Hall (QAH) states. Additionally, rotating the in-plane magnetic orientation of the substrate enables tunability of the Chern number and chirality, within a moderate range of SOC strength. Furthermore, the antiferromagnetic coupling of the magnetic substrate can induce valley-based QAH states with substantial valley gaps, leveraging its high Curie temperature (TC) to enable the realization of multiple tunable magnetic topologies at elevated temperatures. Our findings provide a straightforward strategy for the design and manipulation of spintronic and valleytronic devices that can potentially operate under high-temperature conditions.

cond-mat.mtrl-sci↗

arXiv2024

Decoupling of carbonate-organic carbon isotope during the Carnian Pluvial Episode

The Carnian Pluvial Episode (CPE) was a major global climate change event in the early Late Triassic that significantly affected marine ecosystems and carbon cycles. One of the most prominent features of the CPE is the coupled multiple negative carbonate-organic carbon isotope excursions. However, at Erguan and Xiashulao from eastern Tethys, a decoupling between carbonate-organic carbon isotope during CPE was observed. At the end of early Carnian (Julian), the carbonate carbon isotope showed a negative excursion of 2-3 per-mille, while the organic carbon isotope exhibited a positive excursion of about 3-4 per-mille. In addition, increased terrestrial inputs is indicated by the rising C/N (3 to 10) and decreasing Y/Ho (42 to 27) that coexist with this decoupling. The coupling of carbon isotope negative excursions is from the shallow shelves and the deep slopes, whereas the decoupling occurs from the deep shelf to the shallow slope. In the deep shelf to the shallow slope, sedimentary organic matter is mainly sourced from pelagic before the CPE as evidenced by low C/N (3) and high Y/Ho (36-42). During the CPE, the increased fresh water flux (Sr/Ba <1) enhanced terrestrial input in organic matter, which may cause positive excursions in the carbon isotope record with elevated TOC content. As a result, the carbonate-organic carbon isotope decoupled. In contrast, organic matter in sediments from the shallow shelf and deep slope are mainly from terrestrial and pelagic sources, respectively. This study reveals the significant impact of terrestrial inputs on marine carbon cycling during the Carnian Pluvial Episode, highlighting the crucial role of climate events in modifying the carbon isotope record.

physics.ao-ph↗

arXiv2024

Do Influence Functions Work on Large Language Models?

Influence functions are important for quantifying the impact of individual training data points on a model's predictions. Although extensive research has been conducted on influence functions in traditional machine learning models, their application to large language models (LLMs) has been limited. In this work, we conduct a systematic study to address a key question: do influence functions work on LLMs? Specifically, we evaluate influence functions across multiple tasks and find that they consistently perform poorly in most settings. Our further investigation reveals that their poor performance can be attributed to: (1) inevitable approximation errors when estimating the iHVP component due to the scale of LLMs, (2) uncertain convergence during fine-tuning, and, more fundamentally, (3) the definition itself, as changes in model parameters do not necessarily correlate with changes in LLM behavior. Thus, our study suggests the need for alternative approaches for identifying influential samples.

cs.CL↗

arXiv2024

Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models

Despite significant ongoing efforts in safety alignment, large language models (LLMs) such as GPT-4 and LLaMA 3 remain vulnerable to jailbreak attacks that can induce harmful behaviors, including through the use of adversarial suffixes. Building on prior research, we hypothesize that these adversarial suffixes are not mere bugs but may represent features that can dominate the LLM's behavior. To evaluate this hypothesis, we conduct several experiments. First, we demonstrate that benign features can be effectively made to function as adversarial suffixes, i.e., we develop a feature extraction method to extract sample-agnostic features from benign dataset in the form of suffixes and show that these suffixes may effectively compromise safety alignment. Second, we show that adversarial suffixes generated from jailbreak attacks may contain meaningful features, i.e., appending the same suffix to different prompts results in responses exhibiting specific characteristics. Third, we show that such benign-yet-safety-compromising features can be easily introduced through fine-tuning using only benign datasets. As a result, we are able to completely eliminate GPT's safety alignment in a blackbox setting through finetuning with only benign data. Our code and data is available at \url{https://github.com/suffix-maybe-feature/adver-suffix-maybe-features}.

cs.CR↗

arXiv2024

Bridging the User-side Knowledge Gap in Knowledge-aware Recommendations with Large Language Models

In recent years, knowledge graphs have been integrated into recommender systems as item-side auxiliary information, enhancing recommendation accuracy. However, constructing and integrating structural user-side knowledge remains a significant challenge due to the improper granularity and inherent scarcity of user-side features. Recent advancements in Large Language Models (LLMs) offer the potential to bridge this gap by leveraging their human behavior understanding and extensive real-world knowledge. Nevertheless, integrating LLM-generated information into recommender systems presents challenges, including the risk of noisy information and the need for additional knowledge transfer. In this paper, we propose an LLM-based user-side knowledge inference method alongside a carefully designed recommendation framework to address these challenges. Our approach employs LLMs to infer user interests based on historical behaviors, integrating this user-side information with item-side and collaborative data to construct a hybrid structure: the Collaborative Interest Knowledge Graph (CIKG). Furthermore, we propose a CIKG-based recommendation framework that includes a user interest reconstruction module and a cross-domain contrastive learning module to mitigate potential noise and facilitate knowledge transfer. We conduct extensive experiments on three real-world datasets to validate the effectiveness of our method. Our approach achieves state-of-the-art performance compared to competitive baselines, particularly for users with sparse interactions.

cs.IR↗