arXiv subjects

Hu Xu

Publications and source records attributed to Hu Xu.

At least 37 records · Page 2Linked to original sources

arXiv2025

SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models

We introduce SelfCite, a novel self-supervised approach that aligns LLMs to generate high-quality, fine-grained, sentence-level citations for the statements in their generated responses. Instead of only relying on costly and labor-intensive annotations, SelfCite leverages a reward signal provided by the LLM itself through context ablation: If a citation is necessary, removing the cited text from the context should prevent the same response; if sufficient, retaining the cited text alone should preserve the same response. This reward can guide the inference-time best-of-N sampling strategy to improve citation quality significantly, as well as be used in preference optimization to directly fine-tune the models for generating better citations. The effectiveness of SelfCite is demonstrated by increasing citation F1 up to 5.3 points on the LongBench-Cite benchmark across five long-form question answering tasks. The source code is available at https://github.com/facebookresearch/SelfCite

cs.CL↗

arXiv2025

GM-LDM: Latent Diffusion Model for Brain Biomarker Identification through Functional Data-Driven Gray Matter Synthesis

Generative models based on deep learning have shown significant potential in medical imaging, particularly for modality transformation and multimodal fusion in MRI-based brain imaging. This study introduces GM-LDM, a novel framework that leverages the latent diffusion model (LDM) to enhance the efficiency and precision of MRI generation tasks. GM-LDM integrates a 3D autoencoder, pre-trained on the large-scale ABCD MRI dataset, achieving statistical consistency through KL divergence loss. We employ a Vision Transformer (ViT)-based encoder-decoder as the denoising network to optimize generation quality. The framework flexibly incorporates conditional data, such as functional network connectivity (FNC) data, enabling personalized brain imaging, biomarker identification, and functional-to-structural information translation for brain diseases like schizophrenia.

eess.IV↗

arXiv2025

Modeling and computation of the effective elastic behavior of parallelogram origami metamaterials

Origami metamaterials made of repeating unit cells of parallelogram panels joined at folds dramatically change their shape through a collective motion of their cells. Here we develop an effective elastic model and numerical method to study the large deformation response of these metamaterials under a broad class of loads. The model builds on an effective plate theory derived in our prior work [64]. The theory captures the overall shape change of all slightly stressed parallelogram origami deformations through nonlinear geometric compatibility constraints that couple the origami's (cell averaged) effective deformation to an auxiliary angle field quantifying its cell-by-cell actuation. It also assigns to each such origami deformation a plate energy associated to these effective fields. Seeking a constitutive model that is faithful to the theory but also practical to simulate, we relax the geometric constraints via corresponding elastic energy penalties; we also simplify the plate energy density to embrace its essential character as a regularization to the geometric penalties. The resulting model for parallelogram origami is a generalized elastic continuum that is nonlinear in the effective deformation gradient and angle field and regularized by high-order gradients thereof. We provide a finite element formulation of this model using the $C^0$ interior penalty method to handle second gradients of deformation, and implement it using the open source computing platform Firedrake. We end by using the model and numerical method to study two canonical parallelogram origami patterns, in Miura and Eggbox origami, under a variety of loading conditions.

cond-mat.soft↗

arXiv2025

Perception Encoder: The best visual embeddings are not at the output of the network

We introduce Perception Encoder (PE), a state-of-the-art vision encoder for image and video understanding trained via simple vision-language learning. Traditionally, vision encoders have relied on a variety of pretraining objectives, each tailored to specific downstream tasks such as classification, captioning, or localization. Surprisingly, after scaling our carefully tuned image pretraining recipe and refining with our robust video data engine, we find that contrastive vision-language training alone can produce strong, general embeddings for all of these downstream tasks. There is only one caveat: these embeddings are hidden within the intermediate layers of the network. To draw them out, we introduce two alignment methods: language alignment for multimodal language modeling, and spatial alignment for dense prediction. Together, our PE family of models achieves best-in-class results on a wide variety of tasks, including (1) zero-shot image and video classification and retrieval, simultaneously obtaining 86.6 average zero-shot ImageNet robustness and 76.9 zero-shot Kinetics-400 video classification; (2) document, image, and video Q&A, enabling 94.6 DocVQA, 80.9 InfographicVQA, and 82.7 PerceptionTest with an 8B LLM; and (3) spatial tasks such as detection, tracking, and depth estimation, setting a new COCO state-of-the-art of 66.0 box mAP. To foster further research, we release our models, code, and novel dataset of synthetically and human-annotated videos: https://github.com/facebookresearch/perception_models

cs.CV↗

arXiv2025

General Information Metrics for Improving AI Model Training Efficiency

To address the growing size of AI model training data and the lack of a universal data selection methodology-factors that significantly drive up training costs -- this paper presents the General Information Metrics Evaluation (GIME) method. GIME leverages general information metrics from Objective Information Theory (OIT), including volume, delay, scope, granularity, variety, duration, sampling rate, aggregation, coverage, distortion, and mismatch to optimize dataset selection for training purposes. Comprehensive experiments conducted across diverse domains, such as CTR Prediction, Civil Case Prediction, and Weather Forecasting, demonstrate that GIME effectively preserves model performance while substantially reducing both training time and costs. Additionally, applying GIME within the Judicial AI Program led to a remarkable 39.56% reduction in total model training expenses, underscoring its potential to support efficient and sustainable AI development.

cs.LG↗

arXiv2025

Derivation of an effective plate theory for parallelogram origami from bar and hinge elasticity

Periodic origami patterns made with repeating unit cells of creases and panels bend and twist in complex ways. In principle, such soft modes of deformation admit a simplified asymptotic description in the limit of a large number of cells. Starting from a bar and hinge model for the elastic energy of a generic four parallelogram panel origami pattern, we derive a complete set of geometric compatibility conditions identifying the pattern's soft modes in this limit. The compatibility equations form a system of partial differential equations constraining the actuation of the origami's creases (a scalar angle field) and the relative rotations of its unit cells (a pair of skew tensor fields). We show that every solution of the compatibility equations is the limit of a sequence of soft modes -- origami deformations with finite bending energy and negligible stretching. Using these sequences, we derive a plate-like theory for parallelogram origami patterns with an explicit coarse-grained quadratic energy depending on the gradient of the crease-actuation and the relative rotations of the cells. Finally, we illustrate our theory in the context of two well-known origami designs: the Miura and Eggbox patterns. Though these patterns are distinguished in their anticlastic and synclastic bending responses, they show a universal twisting response. General soft modes captured by our theory involve a rich nonlinear interplay between actuation, bending and twisting, determined by the underlying crease geometry.

cond-mat.soft↗

arXiv2024

Altogether: Image Captioning via Re-aligning Alt-text

This paper focuses on creating synthetic data to improve the quality of image captions. Existing works typically have two shortcomings. First, they caption images from scratch, ignoring existing alt-text metadata, and second, lack transparency if the captioners' training data (e.g. GPT) is unknown. In this paper, we study a principled approach Altogether based on the key idea to edit and re-align existing alt-texts associated with the images. To generate training data, we perform human annotation where annotators start with the existing alt-text and re-align it to the image content in multiple rounds, consequently constructing captions with rich visual concepts. This differs from prior work that carries out human annotation as a one-time description task solely based on images and annotator knowledge. We train a captioner on this data that generalizes the process of re-aligning alt-texts at scale. Our results show our Altogether approach leads to richer image captions that also improve text-to-image generation and zero-shot image classification tasks.

cs.CV↗

arXiv2024

Entropy-driven electron density and effective model Hamiltonian for boron systems

The unique electron deficiency of boron makes it challenging to determine the stable structures, leading to a wide variety of forms. In this work, we introduce a statistical model based on grand canonical ensemble theory that incorporates the octet rule to determine electron density in boron systems. This parameter-free model, referred to as the bonding free energy (BFE) model, aligns well with first-principles calculations and accurately predicts total energies. For borane clusters, the model successfully predicts isomer energies, hydrogen diffusion pathways, and optimal charge quantity for closo-boranes. In all-boron clusters, the absence of B-H bond constraints enables increased electron delocalization and flexibility. The BFE model systematically explains the geometric structures and chemical bonding in boron clusters, revealing variations in electron density that clarify their structural diversity. For borophene, the BFE model predicts that hexagonal vacancy distributions are influenced by bonding entropy, with uniform electron density enhancing stability. Notably, our model predicts borophenes with a vacancy concentration of 1 6 to exhibit increased stability with long-range periodicity. Therefore, the BFE model serves as a practical criterion for structure prediction, providing essential insights into the stability and physical properties of boron-based systems.

cond-mat.mtrl-sci↗

arXiv2024

DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment

Self-supervised visual foundation models produce powerful embeddings that achieve remarkable performance on a wide range of downstream tasks. However, unlike vision-language models such as CLIP, self-supervised visual features are not readily aligned with language, hindering their adoption in open-vocabulary tasks. Our method, named dino.txt, unlocks this new ability for DINOv2, a widely used self-supervised visual encoder. We build upon the LiT training strategy, which trains a text encoder to align with a frozen vision model but leads to unsatisfactory results on dense tasks. We propose several key ingredients to improve performance on both global and dense tasks, such as concatenating the [CLS] token with the patch average to train the alignment and curating data using both text and image modalities. With these, we successfully train a CLIP-like model with only a fraction of the computational cost compared to CLIP while achieving state-of-the-art results in zero-shot classification and open-vocabulary semantic segmentation.

cs.CV↗

arXiv2024

A parameter-free statistical model for two-dimensional carbon nanostructures

Energy degeneracy in physical systems may be induced by symmetries of the Hamiltonian, and the resonance of degeneracy states in carbon nanostructures can effectively enhance the stability of the system. Combining the octet rule, we introduce a parameter-free statistical model to determine the physical properties by lifting the energy degeneracy in carbon nanostructures. This model offers a direct path to accurately ascertain electron density distributions in quantum systems, akin to how charge density is used in density functional theory to deduce system properties. Our methodology diverges from traditional quantum mechanics, focusing instead on this unique statistical model by minimizing bonding free energy to determine the fundamental properties of materials. Applied to carbon nanoclusters and graphynes, our model not only precisely predicts bonding energies and electron density without relying on external parameters, but also enhances the prediction of electronic structures through bond occupancy numbers, which act as effective hopping integrals. This innovation offers insights into the structural properties and quantum behavior of electrons across various dimensions.

cond-mat.mes-hall↗

arXiv2024

The Llama 3 Herd of Models

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

cs.AI↗

arXiv2024

LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding

Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance.

cs.CV↗

arXiv2024

Rethinking the Diffusion Models for Numerical Tabular Data Imputation from the Perspective of Wasserstein Gradient Flow

Diffusion models (DMs) have gained attention in Missing Data Imputation (MDI), but there remain two long-neglected issues to be addressed: (1). Inaccurate Imputation, which arises from inherently sample-diversification-pursuing generative process of DMs. (2). Difficult Training, which stems from intricate design required for the mask matrix in model training stage. To address these concerns within the realm of numerical tabular datasets, we introduce a novel principled approach termed Kernelized Negative Entropy-regularized Wasserstein gradient flow Imputation (KnewImp). Specifically, based on Wasserstein gradient flow (WGF) framework, we first prove that issue (1) stems from the cost functionals implicitly maximized in DM-based MDI are equivalent to the MDI's objective plus diversification-promoting non-negative terms. Based on this, we then design a novel cost functional with diversification-discouraging negative entropy and derive our KnewImp approach within WGF framework and reproducing kernel Hilbert space. After that, we prove that the imputation procedure of KnewImp can be derived from another cost functional related to the joint distribution, eliminating the need for the mask matrix and hence naturally addressing issue (2). Extensive experiments demonstrate that our proposed KnewImp approach significantly outperforms existing state-of-the-art methods.

cs.LG↗

arXiv2024

An Introduction to Vision-Language Modeling

Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.

cs.LG↗

arXiv2024

Text Quality-Based Pruning for Efficient Training of Language Models

In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious. In this paper we propose a novel method for numerically evaluating text quality in large unlabelled NLP datasets in a model agnostic manner to assign the text instances a "quality score". By proposing the text quality metric, the paper establishes a framework to identify and eliminate low-quality text instances, leading to improved training efficiency for LM models. Experimental results over multiple models and datasets demonstrate the efficacy of this approach, showcasing substantial gains in training effectiveness and highlighting the potential for resource-efficient LM training. For example, we observe an absolute accuracy improvement of 0.9% averaged over 14 downstream evaluation tasks for multiple LM models while using 40% lesser data and training 42% faster when training on the OpenWebText dataset and 0.8% average absolute accuracy improvement while using 20% lesser data and training 21% faster on the Wikipedia dataset.

cs.CL↗

arXiv2024

MoDE: CLIP Data Experts via Clustering

The success of contrastive language-image pretraining (CLIP) relies on the supervision from the pairing between images and captions, which tends to be noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn a system of CLIP data experts via clustering. Each data expert is trained on one data cluster, being less sensitive to false negative noises in other clusters. At inference time, we ensemble their outputs by applying weights determined through the correlation between task metadata and cluster conditions. To estimate the correlation precisely, the samples in one cluster should be semantically similar, but the number of data experts should still be reasonable for training and inference. As such, we consider the ontology in human language and propose to use fine-grained cluster centers to represent each data expert at a coarse-grained level. Experimental studies show that four CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and OpenCLIP on zero-shot image classification but with less ($<$35\%) training cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly include new data experts. The code is available at https://github.com/facebookresearch/MetaCLIP/tree/main/mode.

cs.CV↗

arXiv2024

Prediction of topotactic transition from black to blue phosphorus induced by surface Br adsorption

Based on first-principles calculations, we propose a potential access to the yet unrealized freestanding blue phosphorus (blueP) through transformation of black phosphorus (blackP) induced by surface bromine (Br) adsorption. Formation of the Br-P bonds disrupts the original sp3 configurations in blackP, generates unpaired pz electrons and induces a structural transformation that results in blueP formation by re-pairing the pz orbitals. Ab initio molecular dynamics simulations confirm that randomly adsorbed Br adatoms on bilayer blackP spontaneously diffuse into specific patterns to render the emergence of the blueP phase. The expected obtainment Br-passivated blueP nanoribbons exhibit tunable band gaps in a wide range and high carrier mobilities of the order of 1000 cm2V-1s-1. This study provides an opportunity to fabricate blueP through the conversion from blackP by tuning its surface chemistry.

cond-mat.mtrl-sci↗

arXiv2024

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains, such as coding, math reasoning and world knowledge. Our method, named Branch-Train-MiX (BTX), starts from a seed model, which is branched to train experts in embarrassingly parallel fashion with high throughput and reduced communication cost. After individual experts are asynchronously trained, BTX brings together their feedforward parameters as experts in Mixture-of-Expert (MoE) layers and averages the remaining parameters, followed by an MoE-finetuning stage to learn token-level routing. BTX generalizes two special cases, the Branch-Train-Merge method, which does not have the MoE finetuning stage to learn routing, and sparse upcycling, which omits the stage of training experts asynchronously. Compared to alternative approaches, BTX achieves the best accuracy-efficiency tradeoff.

cs.CL↗