arXiv subjects

Yan Wang

Publications and source records attributed to Yan Wang.

At least 505 records · Page 28Linked to original sources

arXiv2024

ProteinBench: A Holistic Evaluation of Protein Foundation Models

Recent years have witnessed a surge in the development of protein foundation models, significantly improving performance in protein prediction and generative tasks ranging from 3D structure prediction and protein design to conformational dynamics. However, the capabilities and limitations associated with these models remain poorly understood due to the absence of a unified evaluation framework. To fill this gap, we introduce ProteinBench, a holistic evaluation framework designed to enhance the transparency of protein foundation models. Our approach consists of three key components: (i) A taxonomic classification of tasks that broadly encompass the main challenges in the protein domain, based on the relationships between different protein modalities; (ii) A multi-metric evaluation approach that assesses performance across four key dimensions: quality, novelty, diversity, and robustness; and (iii) In-depth analyses from various user objectives, providing a holistic view of model performance. Our comprehensive evaluation of protein foundation models reveals several key findings that shed light on their current capabilities and limitations. To promote transparency and facilitate further research, we release the evaluation dataset, code, and a public leaderboard publicly for further analysis and a general modular toolkit. We intend for ProteinBench to be a living benchmark for establishing a standardized, in-depth evaluation framework for protein foundation models, driving their development and application while fostering collaboration within the field.

q-bio.QM↗

arXiv2024

Unleashing the Potentials of Likelihood Composition for Multi-modal Language Models

Model fusing has always been an important topic, especially in an era where large language models (LLM) and multi-modal language models (MLM) with different architectures, parameter sizes and training pipelines, are being created all the time. In this work, we propose a post-hoc framework, aiming at fusing heterogeneous models off-the-shell, which we call \textit{likelihood composition}, and the basic idea is to compose multiple models' likelihood distribution when doing a multi-choice visual-question-answering task. Here the core concept, \textit{likelihood}, is actually the log-probability of the candidate answer. In \textit{likelihood composition}, we introduce some basic operations: \textit{debias}, \textit{highlight}, \textit{majority-vote} and \textit{ensemble}. By combining (composing) these basic elements, we get the mixed composition methods: \textit{mix-composition}. Through conducting comprehensive experiments on 9 VQA datasets and 10 MLMs, we prove the effectiveness of \textit{mix-composition} compared with simple \textit{ensemble} or \textit{majority-vote} methods. In this framework, people can propose new basic composition methods and combine them to get the new mixed composition methods. We hope our proposed \textit{likelihood composition} can provide a new perspective of fusing heterogeneous models and inspire the exploration under this framework.

cs.CL↗

arXiv2024

SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding

3D vision-language grounding, which focuses on aligning language with the 3D physical environment, stands as a cornerstone in the development of embodied agents. In comparison to recent advancements in the 2D domain, grounding language in 3D scenes faces several significant challenges: (i) the inherent complexity of 3D scenes due to the diverse object configurations, their rich attributes, and intricate relationships; (ii) the scarcity of paired 3D vision-language data to support grounded learning; and (iii) the absence of a unified learning framework to distill knowledge from grounded 3D data. In this work, we aim to address these three major challenges in 3D vision-language by examining the potential of systematically upscaling 3D vision-language learning in indoor environments. We introduce the first million-scale 3D vision-language dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising 2.5M vision-language pairs derived from both human annotations and our scalable scene-graph-based generation approach. We demonstrate that this scaling allows for a unified pre-training framework, Grounded Pre-training for Scenes (GPS), for 3D vision-language learning. Through extensive experiments, we showcase the effectiveness of GPS by achieving state-of-the-art performance on all existing 3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is unveiled through zero-shot transfer experiments in the challenging 3D vision-language tasks. Project website: https://scene-verse.github.io.

cs.CV↗

arXiv2024

Protein Conformation Generation via Force-Guided SE(3) Diffusion Models

The conformational landscape of proteins is crucial to understanding their functionality in complex biological processes. Traditional physics-based computational methods, such as molecular dynamics (MD) simulations, suffer from rare event sampling and long equilibration time problems, hindering their applications in general protein systems. Recently, deep generative modeling techniques, especially diffusion models, have been employed to generate novel protein conformations. However, existing score-based diffusion methods cannot properly incorporate important physical prior knowledge to guide the generation process, causing large deviations in the sampled protein conformations from the equilibrium distribution. In this paper, to overcome these limitations, we propose a force-guided SE(3) diffusion model, ConfDiff, for protein conformation generation. By incorporating a force-guided network with a mixture of data-based score models, ConfDiff can generate protein conformations with rich diversity while preserving high fidelity. Experiments on a variety of protein conformation prediction tasks, including 12 fast-folding proteins and the Bovine Pancreatic Trypsin Inhibitor (BPTI), demonstrate that our method surpasses the state-of-the-art method.

q-bio.BM↗

arXiv2024

Generic and Scalable Differential Equation Solver for Quantum Scientific Computing

One of the most important topics in quantum scientific computing is solving differential equations. In this paper, generalized quantum functional expansion (QFE) framework is proposed. In the QFE framework, a functional expansion of solution is encoded into a quantum state and the time evolution of the quantum state is solved with variational quantum simulation (VQS). The quantum functional encoding supports different numerical schemes of functional expansions. The lower bound of the required number of qubits is double logarithm of the inverse error bound in the QFE framework. Furthermore, a new parallel Pauli operation strategy is proposed to significantly improve the scalability of VQS. The number of circuits in VQS is exponentially reduced to only the quadratic order of the number of ansatz parameters. Four example differential equations are solved to demonstrate the generic QFE framework.

quant-ph↗

arXiv2024

Highly coherent grain boundaries induced by local pseudo-mirror symmetry in $β$-Ga2O3

Grain boundaries have extensive influence on the performance of crystal materials. However, the atomic-scale structure and its relation with local and crystallographic symmetries remain elusive in low-symmetry crystals. Herein, we find that the local pseudo-mirror-symmetric atomic layer is the common physical origin of a series of highly coherent grain boundaries in the low-symmetry $β$-Ga2O3 crystal. These include the (100) twin boundary and an emerging series of $(h-1'0'2)/(h+1'0'\bar{2})$ coherent asymmetric grain boundaries (CAGBs). Owing to the local pseudo-mirror symmetry and the special geometric relation of the $β$-Ga2O3 conventional cell, these CAGBs place 80% of the boundary atoms in pseudo-coincident sites, exhibiting high coherence under the coincident-site lattice model. With a combination of density functional theory calculations, Czochralski growth experiment, and atomic-scale characterizations, the structure and stability of the $(002)/(20\bar{2})$-A CAGB are confirmed, with a boundary energy density as low as 0.36 J/m2. This CAGB is responsible for the spontaneous formation of a twinned defect facet at the surface steps during the epitaxy growth of $β$-Ga2O3, warranting a substrate orientation selection rule for $β$-Ga2O3. Through this study, we provide insights into the grain boundary physics in the low-symmetry $β$-Ga2O3 crystal while emphasizing the importance of the local pseudo-symmetries in the low-symmetry crystals.

cond-mat.mtrl-sci↗

arXiv2024

PlainUSR: Chasing Faster ConvNet for Efficient Super-Resolution

Reducing latency is a roaring trend in recent super-resolution (SR) research. While recent progress exploits various convolutional blocks, attention modules, and backbones to unlock the full potentials of the convolutional neural network (ConvNet), achieving real-time performance remains a challenge. To this end, we present PlainUSR, a novel framework incorporating three pertinent modifications to expedite ConvNet for efficient SR. For the convolutional block, we squeeze the lighter but slower MobileNetv3 block into a heavier but faster vanilla convolution by reparameterization tricks to balance memory access and calculations. For the attention module, by modulating input with a regional importance map and gate, we introduce local importance-based attention to realize high-order information interaction within a 1-order attention latency. As to the backbone, we propose a plain U-Net that executes channel-wise discriminate splitting and concatenation. In the experimental phase, PlainUSR exhibits impressively low latency, great scalability, and competitive performance compared to both state-of-the-art latency-oriented and quality-oriented methods. In particular, compared to recent NGswin, the PlainUSR-L is 16.4x faster with competitive performance.

eess.IV↗

arXiv2024

Spectral heat flux redistribution upon interfacial transmission

In nonmetallic crystals, heat is transported by phonons of different frequencies, each contributing differently to the overall heat flux spectrum. In this study, we demonstrate a significant redistribution of heat flux among phonon frequencies when phonons transmit across the interface between dissimilar solids. This redistribution arises from the natural tendency of phononic heat to re-establish the equilibrium distribution characteristic of the material through which it propagates. Remarkably, while the heat flux spectra of dissimilar solids are typically distinct in their bulk forms, they can become nearly identical in superlattices or sandwich structures where the layer thicknesses are smaller than the phonon mean free paths. This phenomenon reflects that the redistribution of heat among phonon frequencies to the equilibrium distribution does not occur instantaneously at the interface, rather it develops over some time and distance.

cond-mat.mes-hall↗

arXiv2024

KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models

The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models. While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks. On the other hand, width pruning retains much of the performance but offers slight speed gains. To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance. Our method uses global perplexity-based analysis to determine the importance ratio for each block and provides multiple strategies to prune non-essential KV channels within blocks. Compared to the original model, KVPruner reduces runtime memory usage by 50% and boosts throughput by over 35%. Additionally, our method requires only two hours of LoRA fine-tuning on small datasets to recover most of the performance.

cs.CL↗

arXiv2024

Dynamics of Small Solid Particles on Substrates of Arbitrary Topography

We study the dynamics of a small solid particle arising from the dewetting of a thin film on a curved substrate driven by capillarity, where mass transport is controlled by surface diffusion. We consider the case when the size of the deposited particle is much smaller than the local radius of curvature of the substrate surface. The application of the Onsager variational principle leads to a reduced-order model for the dynamic behaviour of particles on arbitrarily curved substrates. We demonstrate that particles move toward region of the substrate surface with lower mean curvature with a determined velocity. In particular, the velocity is proportional to the substrate curvature gradient and inversely proportional to the size of the particle, with a coefficient that depends on material properties that include the surface energy, surface diffusivity, density, and Young's (wetting) angle. The reduced model is validated by comparing with numerical results for the full, sharp-interface model in both two and three dimensions.

cond-mat.mtrl-sci↗

arXiv2024

A Causal Explainable Guardrails for Large Language Models

Large Language Models (LLMs) have shown impressive performance in natural language tasks, but their outputs can exhibit undesirable attributes or biases. Existing methods for steering LLMs toward desired attributes often assume unbiased representations and rely solely on steering prompts. However, the representations learned from pre-training can introduce semantic biases that influence the steering process, leading to suboptimal results. We propose LLMGuardrail, a novel framework that incorporates causal analysis and adversarial learning to obtain unbiased steering representations in LLMs. LLMGuardrail systematically identifies and blocks the confounding effects of biases, enabling the extraction of unbiased steering representations. Additionally, it includes an explainable component that provides insights into the alignment between the generated output and the desired direction. Experiments demonstrate LLMGuardrail's effectiveness in steering LLMs toward desired attributes while mitigating biases. Our work contributes to the development of safe and reliable LLMs that align with desired attributes.

cs.CL↗

arXiv2024

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models

Large language models (LLMs) face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. Meanwhile, extending the context window in LLMs through post-pretraining is highly resource-intensive. To address this, we introduce LongRecipe, an efficient training strategy for extending the context window of LLMs, including impactful token analysis, position index transformation, and training optimization strategies. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model's understanding of long-range dependencies. Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size, and reduces computational training resource over 85% compared to full sequence training. Furthermore, LongRecipe also preserves the original LLM's capabilities in general tasks. Ultimately, we can extend the effective context window of open-source LLMs from 8k to 128k, achieving performance close to GPT-4 with just one day of dedicated training using a single GPU with 80G memory. Our code is released at https://github.com/zhiyuanhubj/LongRecipe.

cs.CL↗

arXiv2024

Significantly Enhanced Interfacial Thermal Transport between Single-layer Graphene and Water Through Basal-plane Oxidation

Heat transfer between graphene and water is pivotal for various applications, including solarthermal vapor generation and the advanced manufacturing of graphene-based hierarchical structures in solution. In this study, we employ a deep-neural network potential derived from ab initio molecular dynamics to conduct extensive simulations of single-layer graphenewater systems with different levels of oxidation (carbon/oxygen ratio) of the graphene layer. Remarkably, our findings reveal a one-order-of-magnitude enhancement in heat transfer upon oxidizing graphene with hydroxyl or epoxide groups at the graphene surface, underscoring the significant tunability of heat transfer within this system. Given the same oxidation ratio, more dispersed locations of functional groups on graphene surface leads to faster heat dissipation to water.

cond-mat.mtrl-sci↗

arXiv2024

BTMuda: A Bi-level Multi-source unsupervised domain adaptation framework for breast cancer diagnosis

Deep learning has revolutionized the early detection of breast cancer, resulting in a significant decrease in mortality rates. However, difficulties in obtaining annotations and huge variations in distribution between training sets and real scenes have limited their clinical applications. To address these limitations, unsupervised domain adaptation (UDA) methods have been used to transfer knowledge from one labeled source domain to the unlabeled target domain, yet these approaches suffer from severe domain shift issues and often ignore the potential benefits of leveraging multiple relevant sources in practical applications. To address these limitations, in this work, we construct a Three-Branch Mixed extractor and propose a Bi-level Multi-source unsupervised domain adaptation method called BTMuda for breast cancer diagnosis. Our method addresses the problems of domain shift by dividing domain shift issues into two levels: intra-domain and inter-domain. To reduce the intra-domain shift, we jointly train a CNN and a Transformer as two paths of a domain mixed feature extractor to obtain robust representations rich in both low-level local and high-level global information. As for the inter-domain shift, we redesign the Transformer delicately to a three-branch architecture with cross-attention and distillation, which learns domain-invariant representations from multiple domains. Besides, we introduce two alignment modules - one for feature alignment and one for classifier alignment - to improve the alignment process. Extensive experiments conducted on three public mammographic datasets demonstrate that our BTMuda outperforms state-of-the-art methods.

cs.CV↗

arXiv2024

CogVLM2: Visual Language Models for Image and Video Understanding

Beginning with VisualGLM and CogVLM, we are continuously exploring VLMs in pursuit of enhanced vision-language fusion, efficient higher-resolution architecture, and broader modalities and applications. Here we propose the CogVLM2 family, a new generation of visual language models for image and video understanding including CogVLM2, CogVLM2-Video and GLM-4V. As an image understanding model, CogVLM2 inherits the visual expert architecture with improved training recipes in both pre-training and post-training stages, supporting input resolution up to $1344 \times 1344$ pixels. As a video understanding model, CogVLM2-Video integrates multi-frame input with timestamps and proposes automated temporal grounding data construction. Notably, CogVLM2 family has achieved state-of-the-art results on benchmarks like MMBench, MM-Vet, TextVQA, MVBench and VCGBench. All models are open-sourced in https://github.com/THUDM/CogVLM2 and https://github.com/THUDM/GLM-4, contributing to the advancement of the field.

cs.CV↗

arXiv2024

A Survey on Facial Expression Recognition of Static and Dynamic Emotions

Facial expression recognition (FER) aims to analyze emotional states from static images and dynamic sequences, which is pivotal in enhancing anthropomorphic communication among humans, robots, and digital avatars by leveraging AI technologies. As the FER field evolves from controlled laboratory environments to more complex in-the-wild scenarios, advanced methods have been rapidly developed and new challenges and apporaches are encounted, which are not well addressed in existing reviews of FER. This paper offers a comprehensive survey of both image-based static FER (SFER) and video-based dynamic FER (DFER) methods, analyzing from model-oriented development to challenge-focused categorization. We begin with a critical comparison of recent reviews, an introduction to common datasets and evaluation criteria, and an in-depth workflow on FER to establish a robust research foundation. We then systematically review representative approaches addressing eight main challenges in SFER (such as expression disturbance, uncertainties, compound emotions, and cross-domain inconsistency) as well as seven main challenges in DFER (such as key frame sampling, expression intensity variations, and cross-modal alignment). Additionally, we analyze recent advancements, benchmark performances, major applications, and ethical considerations. Finally, we propose five promising future directions and development trends to guide ongoing research. The project page for this paper can be found at https://github.com/wangyanckxx/SurveyFER.

cs.CV↗

arXiv2024

ARANet: Attention-based Residual Adversarial Network with Deep Supervision for Radiotherapy Dose Prediction of Cervical Cancer

Radiation therapy is the mainstay treatment for cervical cancer, and its ultimate goal is to ensure the planning target volume (PTV) reaches the prescribed dose while reducing dose deposition of organs-at-risk (OARs) as much as possible. To achieve these clinical requirements, the medical physicist needs to manually tweak the radiotherapy plan repeatedly in a trial-anderror manner until finding the optimal one in the clinic. However, such trial-and-error processes are quite time-consuming, and the quality of plans highly depends on the experience of the medical physicist. In this paper, we propose an end-to-end Attentionbased Residual Adversarial Network with deep supervision, namely ARANet, to automatically predict the 3D dose distribution of cervical cancer. Specifically, given the computer tomography (CT) images and their corresponding segmentation masks of PTV and OARs, ARANet employs a prediction network to generate the dose maps. We also utilize a multi-scale residual attention module and deep supervision mechanism to enforce the prediction network to extract more valuable dose features while suppressing irrelevant information. Our proposed method is validated on an in-house dataset including 54 cervical cancer patients, and experimental results have demonstrated its obvious superiority compared to other state-of-the-art methods.

cs.CV↗

arXiv2024

Alleviating Class Imbalance in Semi-supervised Multi-organ Segmentation via Balanced Subclass Regularization

Semi-supervised learning (SSL) has shown notable potential in relieving the heavy demand of dense prediction tasks on large-scale well-annotated datasets, especially for the challenging multi-organ segmentation (MoS). However, the prevailing class-imbalance problem in MoS, caused by the substantial variations in organ size, exacerbates the learning difficulty of the SSL network. To alleviate this issue, we present a two-phase semi-supervised network (BSR-Net) with balanced subclass regularization for MoS. Concretely, in Phase I, we introduce a class-balanced subclass generation strategy based on balanced clustering to effectively generate multiple balanced subclasses from original biased ones according to their pixel proportions. Then, in Phase II, we design an auxiliary subclass segmentation (SCS) task within the multi-task framework of the main MoS task. The SCS task contributes a balanced subclass regularization to the main MoS task and transfers unbiased knowledge to the MoS network, thus alleviating the influence of the class-imbalance problem. Extensive experiments conducted on two publicly available datasets, i.e., the MICCAI FLARE 2022 dataset and the WORD dataset, verify the superior performance of our method compared with other methods.

cs.CV↗