Searcharxiv⌕ Search

arXiv subjects

Lijun Li

Publications and source records attributed to Lijun Li.

At least 73 records · Page 4Linked to original sources

Assessment of Multimodal Large Language Models in Alignment with Human Values

Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh). However, in terms of Multimodal Large Language Models (MLLMs), despite their commendable performance in perception and reasoning tasks, their alignment with human values remains largely unexplored, given the complexity of defining hhh dimensions in the visual world and the difficulty in collecting relevant data that accurately mirrors real-world situations. To address this gap, we introduce Ch3Ef, a Compreh3ensive Evaluation dataset and strategy for assessing alignment with human expectations. Ch3Ef dataset contains 1002 human-annotated data samples, covering 12 domains and 46 tasks based on the hhh principle. We also present a unified evaluation strategy supporting assessment across various scenarios and different perspectives. Based on the evaluation results, we summarize over 10 key findings that deepen the understanding of MLLM capabilities, limitations, and the dynamic relationships between evaluation levels, guiding future advancements in the field.

cs.CV↗

EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models

Jailbreak attacks are crucial for identifying and mitigating the security vulnerabilities of Large Language Models (LLMs). They are designed to bypass safeguards and elicit prohibited outputs. However, due to significant differences among various jailbreak methods, there is no standard implementation framework available for the community, which limits comprehensive security evaluations. This paper introduces EasyJailbreak, a unified framework simplifying the construction and evaluation of jailbreak attacks against LLMs. It builds jailbreak attacks using four components: Selector, Mutator, Constraint, and Evaluator. This modular framework enables researchers to easily construct attacks from combinations of novel and existing components. So far, EasyJailbreak supports 11 distinct jailbreak methods and facilitates the security validation of a broad spectrum of LLMs. Our validation across 10 distinct LLMs reveals a significant vulnerability, with an average breach probability of 60% under various jailbreaking attacks. Notably, even advanced models like GPT-3.5-Turbo and GPT-4 exhibit average Attack Success Rates (ASR) of 57% and 33%, respectively. We have released a wealth of resources for researchers, including a web platform, PyPI published package, screencast video, and experimental outputs.

cs.CL↗

From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities

Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents. However, there is still a wide gap between the performance of recent MLLM-based applications and the expectation of the broad public, even though the most powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper strives to enhance understanding of the gap through the lens of a qualitative study on the generalizability, trustworthiness, and causal reasoning capabilities of recent proprietary and open-source MLLMs across four modalities: ie, text, code, image, and video, ultimately aiming to improve the transparency of MLLMs. We believe these properties are several representative factors that define the reliability of MLLMs, in supporting various downstream applications. To be specific, we evaluate the closed-source GPT-4 and Gemini and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed cases, where the qualitative results are then summarized into 12 scores (ie, 4 modalities times 3 properties). In total, we uncover 14 empirical findings that are useful to understand the capabilities and limitations of both proprietary and open-source MLLMs, towards more reliable downstream multi-modal applications.

cs.CV↗

NP-RDMA: Using Commodity RDMA without Pinning Memory

Remote Direct Memory Access (RDMA) has been haunted by the need of pinning down memory regions. Pinning limits the memory utilization because it impedes on-demand paging and swapping. It also increases the initialization latency of large memory applications from seconds to minutes. To remove memory pining, existing approaches often require special hardware which supports page fault, and still have inferior performance. We propose NP-RDMA, which removes memory pinning during memory registration and enables dynamic page fault handling with commodity RDMA NICs. NP-RDMA does not require NICs to support page fault. Instead, by monitoring local memory paging and swapping with MMU-notifier, combining with IOMMU/SMMU-based address mapping, NP-RDMA efficiently detects and handles page fault in the software with near-zero additional latency to non-page-fault RDMA verbs. We implement an LD_PRELOAD library (with a modified kernel module), which is fully compatible with existing RDMA applications. Experiments show that NP-RDMA adds only 0.1{\sim}2 μs latency under non-page-fault scenarios. Moreover, NP-RDMA adds only 3.5{\sim}5.7 μs and 60 μs under minor or major page faults, respectively, which is 500x faster than ODP which uses advanced NICs that support page fault. With non-pinned memory, Spark initialization is 20x faster and the physical memory usage reduces by 86% with only 5.4% slowdown. Enterprise storage can expand to 5x capacity with SSDs while the average latency is only 10% higher. To the best of our knowledge, NP-RDMA is the first efficient and application-transparent software approach to remove memory pinning using commodity RDMA NICs.

cs.NI↗

RenderIH: A Large-scale Synthetic Dataset for 3D Interacting Hand Pose Estimation

The current interacting hand (IH) datasets are relatively simplistic in terms of background and texture, with hand joints being annotated by a machine annotator, which may result in inaccuracies, and the diversity of pose distribution is limited. However, the variability of background, pose distribution, and texture can greatly influence the generalization ability. Therefore, we present a large-scale synthetic dataset RenderIH for interacting hands with accurate and diverse pose annotations. The dataset contains 1M photo-realistic images with varied backgrounds, perspectives, and hand textures. To generate natural and diverse interacting poses, we propose a new pose optimization algorithm. Additionally, for better pose estimation accuracy, we introduce a transformer-based pose estimation network, TransHand, to leverage the correlation between interacting hands and verify the effectiveness of RenderIH in improving results. Our dataset is model-agnostic and can improve more accuracy of any hand pose estimation method in comparison to other real or synthetic datasets. Experiments have shown that pretraining on our synthetic data can significantly decrease the error from 6.76mm to 5.79mm, and our Transhand surpasses contemporary methods. Our dataset and code are available at https://github.com/adwardlee/RenderIH.

cs.CV↗

DiffHand: End-to-End Hand Mesh Reconstruction via Diffusion Models

Hand mesh reconstruction from the monocular image is a challenging task due to its depth ambiguity and severe occlusion, there remains a non-unique mapping between the monocular image and hand mesh. To address this, we develop DiffHand, the first diffusion-based framework that approaches hand mesh reconstruction as a denoising diffusion process. Our one-stage pipeline utilizes noise to model the uncertainty distribution of the intermediate hand mesh in a forward process. We reformulate the denoising diffusion process to gradually refine noisy hand mesh and then select mesh with the highest probability of being correct based on the image itself, rather than relying on 2D joints extracted beforehand. To better model the connectivity of hand vertices, we design a novel network module called the cross-modality decoder. Extensive experiments on the popular benchmarks demonstrate that our method outperforms the state-of-the-art hand mesh reconstruction approaches by achieving 5.8mm PA-MPJPE on the Freihand test set, 4.98mm PA-MPJPE on the DexYCB test set.

cs.CV↗

One-stage Action Detection Transformer

In this work, we introduce our solution to the EPIC-KITCHENS-100 2022 Action Detection challenge. One-stage Action Detection Transformer (OADT) is proposed to model the temporal connection of video segments. With the help of OADT, both the category and time boundary can be recognized simultaneously. After ensembling multiple OADT models trained from different features, our model can reach 21.28\% action mAP and ranks the 1st on the test-set of the Action detection challenge.

cs.CV↗

Approximate Group Fairness for Clustering

We incorporate group fairness into the algorithmic centroid clustering problem, where $k$ centers are to be located to serve $n$ agents distributed in a metric space. We refine the notion of proportional fairness proposed in [Chen et al., ICML 2019] as {\em core fairness}, and $k$-clustering is in the core if no coalition containing at least $n/k$ agents can strictly decrease their total distance by deviating to a new center together. Our solution concept is motivated by the situation where agents are able to coordinate and utilities are transferable. A string of existence, hardness and approximability results is provided. Particularly, we propose two dimensions to relax core requirements: one is on the degree of distance improvement, and the other is on the size of deviating coalition. For both relaxations and their combination, we study the extent to which relaxed core fairness can be satisfied in metric spaces including line, tree and general metric space, and design approximation algorithms accordingly.

cs.GT↗

Gate-Tunable Reversible Rashba-Edelstein Effect in a Few-Layer Graphene/2H-TaS2 Heterostructure at Room Temperature

We report the observation of current-induced spin polarization, the Rashba-Edelstein effect (REE), and its Onsager reciprocal phenomenon, the spin galvanic effect (SGE), in a few-layer graphene/2H-TaS2 heterostructure at room temperature. Spin-sensitive electrical measurements unveil full spin-polarization reversal by an applied gate voltage. The observed gate-tunable charge-to-spin conversion is explained by the ideal work function mismatch between 2H-TaS2 and graphene, which allows strong interface-induced Bychkov-Rashba interaction with a spin-gap reaching 70 meV, while keeping the Dirac nature of the spectrum intact across electron and hole sectors. The reversible electrical generation and control of the nonequilibrium spin polarization vector, not previously observed in a nonmagnetic material, are elegant manifestations of emergent 2D Dirac fermions with robust spin-helical structure. Our experimental findings, supported by first-principles relativistic electronic structure and transport calculations, demonstrate a route to design low-power spin-logic circuits from layered materials.

cond-mat.mes-hall↗

NATTACK: Learning the Distributions of Adversarial Examples for an Improved Black-Box Attack on Deep Neural Networks

Powerful adversarial attack methods are vital for understanding how to construct robust deep neural networks (DNNs) and for thoroughly testing defense techniques. In this paper, we propose a black-box adversarial attack algorithm that can defeat both vanilla DNNs and those generated by various defense techniques developed recently. Instead of searching for an "optimal" adversarial example for a benign input to a targeted DNN, our algorithm finds a probability density distribution over a small region centered around the input, such that a sample drawn from this distribution is likely an adversarial example, without the need of accessing the DNN's internal layers or weights. Our approach is universal as it can successfully attack different neural networks by a single algorithm. It is also strong; according to the testing against 2 vanilla DNNs and 13 defended ones, it outperforms state-of-the-art black-box or white-box attack methods for most test cases. Additionally, our results reveal that adversarial training remains one of the best defense techniques, and the adversarial examples are not as transferable across defended DNNs as them across vanilla DNNs.

cs.LG↗

End-to-End Video Captioning with Multitask Reinforcement Learning

Although end-to-end (E2E) learning has led to impressive progress on a variety of visual understanding tasks, it is often impeded by hardware constraints (e.g., GPU memory) and is prone to overfitting. When it comes to video captioning, one of the most challenging benchmark tasks in computer vision, those limitations of E2E learning are especially amplified by the fact that both the input videos and output captions are lengthy sequences. Indeed, state-of-the-art methods for video captioning process video frames by convolutional neural networks and generate captions by unrolling recurrent neural networks. If we connect them in an E2E manner, the resulting model is both memory-consuming and data-hungry, making it extremely hard to train. In this paper, we propose a multitask reinforcement learning approach to training an E2E video captioning model. The main idea is to mine and construct as many effective tasks (e.g., attributes, rewards, and the captions) as possible from the human captioned videos such that they can jointly regulate the search space of the E2E neural network, from which an E2E video captioning model can be found and generalized to the testing phase. To the best of our knowledge, this is the first video captioning model that is trained end-to-end from the raw video input to the caption output. Experimental results show that such a model outperforms existing ones to a large margin on two benchmark video captioning datasets.

cs.CV↗

Superconducting Order from Disorder in 2H-TaSe$_{2-x}$S$_{x}$ (0$\leq$x$\leq$2)

We report on the emergence of robust superconducting order in single crystal alloys of 2H-TaSe$_{2-x}$S$_{x}$ (0$\leq$x$\leq$2) . The critical temperature of the alloy is surprisingly higher than that of the two end compounds TaSe$_{2}$ and TaS$_{2}$. The evolution of superconducting critical temperature T$_{c} (x)$ correlates with the full width at half maximum of the Bragg peaks and with the linear term of the high temperature resistivity. The conductivity of the crystals near the middle of the alloy series is higher or similar than that of either one of the end members 2H-TaSe$_{2}$ and/or 2H-TaS$_{2}$. It is known that in these materials superconductivity (SC) is in close competition with charge density wave (CDW) order. We interpret our experimental findings in a picture where disorder tilts this balance in favor of superconductivity by destroying the CDW order.

cond-mat.supr-con↗

Model-Checking of Linear-Time Properties Based on Possibility Measure

We study the LTL model-checking in possibilistic Kripke structure using possibility measure. First, the notion of possibilistic Kripke structure and the related possibility measure are introduced, then model-checking of reachability and repeated reachability linear-time properties in finite possibilistic Kripke structure are studied. Standard safety property and -regular property in possibilistic Kripke structure are introduced, the verification of regular safety property and -regular property using finite automata are thoroughly studied. It has been shown that the verification of regular safety property and -regular property in finite possibilistic Kripke structure can be transformed into the verification of reachability property and repeated reachability property in the product possibilistic Kripke structure introduced in this paper. Several examples are given to illustrate the methods presented in the paper.

cs.LO↗

Superconductivity and Charge Density Wave in ZrTe$_{3-x}$Se$_{x}$

Charge density wave (CDW), the periodic modulation of the electronic charge density, will open a gap on the Fermi surface that commonly leads to decreased or vanishing conductivity. On the other hand superconductivity, a commonly believed competing order, features a Fermi surface gap that results in infinite conductivity. Here we report that superconductivity emerges upon Se doping in CDW conductor ZrTe$_{3}$ when the long range CDW order is gradually suppressed. Superconducting critical temperature $T_c(x)$ in ZrTe$_{3-x}$Se$_x$ (${0\leq}x\leq0.1$) increases up to 4 K plateau for $0.04$$\leq$$x$$\leq$$0.07$. Further increase in Se content results in diminishing $T_{c}$ and filametary superconductivity. The CDW modes from Raman spectra are observed in $x$ = 0.04 and 0.1 crystals, where signature of ZrTe$_{3}$ CDW order in resistivity vanishes. The electronic-scattering for high $T_{c}$ crystals is dominated by local CDW fluctuations at high temperures, the resistivity is linear up to highest measured $T=300K$ and contributes to substantial in-plane anisotropy.

cond-mat.supr-con↗

Electron-hole asymmetry, Dirac fermions, and quantum magnetoresistance in BaMnBi2

We report two-dimensional quantum transport and Dirac fermions in BaMnBi2 single crystals. BaMnBi2 is a layered bad metal with highly anisotropic conductivity and magnetic order below 290 K. Magnetotransport properties, nonzero Berry phase, small cyclotronmass, and the first-principles band structure calculations indicate the presence of Dirac fermions in Bi square nets. Quantum oscillations in the Hall channel suggest the presence of both electron and hole pockets, whereas Dirac and parabolic states coexist at the Fermi level.

cond-mat.mtrl-sci↗

Anisotropic giant magnetoresistance in NbSb2

The extremely large transverse magnetoreistance (the magnetoresistant ratio $\sim 1.3\times10^5\%$ in 2 K and 9 T field, and $4.3\times 10^6\%$ in 0.4 K and 32 T field, without saturation), and the metal-semiconductor crossover induced by magnetic field, are reported in NbSb$_2$ single crystal with electric current parallel to the $b$-axis. The metal-semiconductor crossover is preserved when the current is along the $ac$-plane but the magnetoresistant ratio is significantly suppressed. The sign reversal of the Hall resistivity in the field close to the crossover point, and the electronic structure calculation reveals the coexistence of a small number of holes with very high mobility and a large number of electrons with low mobility. These effects are attributed to the change of the Fermi surface induced by the magnetic field.

cond-mat.mtrl-sci↗

Microscopic evidence for strong periodic lattice distortion in 2D charge-density wave systems

In the quasi-2D electron systems of the layered transition metal dichalcogenides (TMD) there is still a controversy about the nature of the transitions to charge-density wave (CDW) phases, i.e. whether they are described by a Peierls-type mechanism or by a lattice-driven model. By performing scanning tunneling microscopy (STM) experiments on the canonical TMD-CDW systems, we have imaged the electronic modulation and the lattice distortion separately in 2H-TaS$_2$, TaSe$_2$, and NbSe$_2$. Across the three materials, we found dominant lattice contributions instead of the electronic modulation expected from Peierls transitions, in contrast to the CDW states that show the hallmark of contrast inversion between filled and empty states. Our results imply that the periodic lattice distortion (PLD) plays a vital role in the formation of CDW phases in the TMDs and illustrate the importance of taking into account the more complicated lattice degree of freedom when studying correlated electron systems.

cond-mat.str-el↗

New Superconductivity in Layered 1T-TaS2-xSex Single Crystals Fabricated by Chemical Vapor Transport

Layered transition-metal dichalcogenides 1T-TaS2-xSex (0<=x<=2) single crystals have been successfully fabricated by using a chemical vapor transport technique in which Ta locates in octahedral coordination with S and Se atoms. This is the first superconducting example by the substitution of S site, which violates an initial rule based on the fact that superconductivity merely emerges in 1T-TaS2 by applying the high pressure or substitution of Ta site. We demonstrate the appearance of a series of electronic states in 1T-TaS2-xSex with Se content. Namely, the Mott phase melts into a nearly commensurate charge-density-wave (NCCDW) phase, superconductivity in a wide x range develops within the NCCDW state, and finally commensurate charge-density-wave (CCDW) phase reproduces for heavy Se content. The present results reveal that superconductivity is only characterized by robust Ta 5d band, demonstrating the universal nature in 1T-TaS2 systems that superconductivity and NCCDW phase coexist in the real space.

cond-mat.supr-con↗