arXiv subjects

Jiachen Li

Publications and source records attributed to Jiachen Li.

At least 145 records · Page 8Linked to original sources

arXiv2024

Interacting-bath dynamical embedding for capturing non-local electron correlation in solids

Quantitative simulation of electronic structure of solids requires treating local and non-local electron correlations on an equal footing. We present a new ab initio formulation of Green's function embedding which, unlike dynamical mean-field theory that uses non-interacting bath, derives bath representation with general two-particle interactions in a systematically improvable manner. The resulting interacting-bath dynamical embedding theory (ibDET) utilizes an efficient real-axis coupled-cluster solver to compute the self-energy, approaching the full system limit at much reduced cost. When combined with the GW theory, GW+ibDET achieves good agreement with experimental spectral properties across a range of semiconducting, insulating and metallic materials. Our approach also enables quantifying the role of non-local electron correlation in determining material properties and addressing the long-standing debate on the bandwidth narrowing of metallic sodium.

cond-mat.mtrl-sci↗

arXiv2024

Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation

Recent studies show that text-to-image (T2I) models are vulnerable to adversarial attacks, especially with noun perturbations in text prompts. In this study, we investigate the impact of adversarial attacks on different POS tags within text prompts on the images generated by T2I models. We create a high-quality dataset for realistic POS tag token swapping and perform gradient-based attacks to find adversarial suffixes that mislead T2I models into generating images with altered tokens. Our empirical results show that the attack success rate (ASR) varies significantly among different POS tag categories, with nouns, proper nouns, and adjectives being the easiest to attack. We explore the mechanism behind the steering effect of adversarial suffixes, finding that the number of critical tokens and content fusion vary among POS tags, while features like suffix transferability are consistent across categories. We have made our implementation publicly available at - https://github.com/shahariar-shibli/Adversarial-Attack-on-POS-Tags.

cs.CL↗

arXiv2024

CoMamba: Real-time Cooperative Perception Unlocked with State Space Models

Cooperative perception systems play a vital role in enhancing the safety and efficiency of vehicular autonomy. Although recent studies have highlighted the efficacy of vehicle-to-everything (V2X) communication techniques in autonomous driving, a significant challenge persists: how to efficiently integrate multiple high-bandwidth features across an expanding network of connected agents such as vehicles and infrastructure. In this paper, we introduce CoMamba, a novel cooperative 3D detection framework designed to leverage state-space models for real-time onboard vehicle perception. Compared to prior state-of-the-art transformer-based models, CoMamba enjoys being a more scalable 3D model using bidirectional state space models, bypassing the quadratic complexity pain-point of attention mechanisms. Through extensive experimentation on V2X/V2V datasets, CoMamba achieves superior performance compared to existing methods while maintaining real-time processing capabilities. The proposed framework not only enhances object detection accuracy but also significantly reduces processing time, making it a promising solution for next-generation cooperative perception systems in intelligent transportation networks.

cs.CV↗

arXiv2024

Human and LLM-Based Voice Assistant Interaction: An Analytical Framework for User Verbal and Nonverbal Behaviors

Recent progress in large language model (LLM) technology has significantly enhanced the interaction experience between humans and voice assistants (VAs). This project aims to explore a user's continuous interaction with LLM-based VA (LLM-VA) during a complex task. We recruited 12 participants to interact with an LLM-VA during a cooking task, selected for its complexity and the requirement for continuous interaction. We observed that users show both verbal and nonverbal behaviors, though they know that the LLM-VA can not capture those nonverbal signals. Despite the prevalence of nonverbal behavior in human-human communication, there is no established analytical methodology or framework for exploring it in human-VA interactions. After analyzing 3 hours and 39 minutes of video recordings, we developed an analytical framework with three dimensions: 1) behavior characteristics, including both verbal and nonverbal behaviors, 2) interaction stages--exploration, conflict, and integration--that illustrate the progression of user interactions, and 3) stage transition throughout the task. This analytical framework identifies key verbal and nonverbal behaviors that provide a foundation for future research and practical applications in optimizing human and LLM-VA interactions.

cs.HC↗

arXiv2024

Navigating the Paradox: Challenges and Strategies of University Students Managing Mental Health Medication in Real-World Practices

Mental health has become a growing concern among university students. While medication is a common treatment, understanding how university students manage their medication for mental health symptoms in real-world practice has not been fully explored. In this study, we conducted semi-structured interviews with university students to understand the unique challenges in the mental health medication management process and their coping strategies, particularly examining the role of various technologies in this process. We discovered that due to struggles with self-acceptance and the interdependent relationship between medication, symptoms, schedules, and life changes, the medication management process for students was a highly dynamic journey involving frequent dosage changes. Thus, students adopted flexible strategies of using minimal technology to manage their medication in different situations while maintaining a high degree of autonomy. Based on our findings, we propose design implications for future technologies to seamlessly integrate into their daily lives and assist students in managing their mental health medications.

cs.HC↗

arXiv2024

MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos

Multimodal Language Language Models (MLLMs) demonstrate the emerging abilities of "world models" -- interpreting and reasoning about complex real-world dynamics. To assess these abilities, we posit videos are the ideal medium, as they encapsulate rich representations of real-world dynamics and causalities. To this end, we introduce MMWorld, a new benchmark for multi-discipline, multi-faceted multimodal video understanding. MMWorld distinguishes itself from previous video understanding benchmarks with two unique advantages: (1) multi-discipline, covering various disciplines that often require domain expertise for comprehensive understanding; (2) multi-faceted reasoning, including explanation, counterfactual thinking, future prediction, etc. MMWorld consists of a human-annotated dataset to evaluate MLLMs with questions about the whole videos and a synthetic dataset to analyze MLLMs within a single modality of perception. Together, MMWorld encompasses 1,910 videos across seven broad disciplines and 69 subdisciplines, complete with 6,627 question-answer pairs and associated captions. The evaluation includes 2 proprietary and 10 open-source MLLMs, which struggle on MMWorld (e.g., GPT-4V performs the best with only 52.3\% accuracy), showing large room for improvement. Further ablation studies reveal other interesting findings such as models' different skill sets from humans. We hope MMWorld can serve as an essential step towards world model evaluation in videos.

cs.CV↗

arXiv2024

Backdoor Attacks against Hybrid Classical-Quantum Neural Networks

Hybrid Quantum Neural Networks (HQNNs) represent a promising advancement in Quantum Machine Learning (QML), yet their security has been rarely explored. In this paper, we present the first systematic study of backdoor attacks on HQNNs. We begin by proposing an attack framework and providing a theoretical analysis of the generalization bounds and minimum perturbation requirements for backdoor attacks on HQNNs. Next, we employ two classic backdoor attack methods on HQNNs and Convolutional Neural Networks (CNNs) to further investigate the robustness of HQNNs. Our experimental results demonstrate that HQNNs are more robust than CNNs, requiring more significant image modifications for successful attacks. Additionally, we introduce the Qcolor backdoor, which utilizes color shifts as triggers and employs the Non-dominated Sorting Genetic Algorithm II (NSGA-II) to optimize hyperparameters. Through extensive experiments, we demonstrate the effectiveness, stealthiness, and robustness of the Qcolor backdoor.

cs.CR↗

arXiv2024

Particle-Particle Random Phase Approximation for Predicting Correlated Excited States of Point Defects

The particle-particle random phase approximation (ppRPA) within the hole-hole channel was recently proposed as an efficient tool for computing excitation energies of point defects in solids [J. Phys. Chem. Lett. 2024, 15, 2757-2764]. In this work, we investigate the application of ppRPA within the particle-particle channel for predicting correlated excited states of point defects, including the carbon-vacancy (VC) in diamond, the oxygen-vacancy (VO) in magnesium oxide (MgO), and the carbon dimer defect (C$_{\text{B}}$C$_{\text{N}}$) in two-dimensional hexagonal boron nitride (h-BN). Starting from a density functional theory calculation of the ($N-2$)-electron ground state, vertical excitation energies of the $N$-electron system are obtained as the differences between the two-electron addition energies. We show that active-space ppRPA with the B3LYP functional yields accurate excitation energies, with errors mostly smaller than 0.1 eV for tested systems compared to available experimental values. We further develop a natural transition orbital scheme within ppRPA, which provides insights into the multireference character of defect states. This study, together with our previous work, establishes ppRPA as a low-cost and accurate method for investigating excited-state properties of point defect systems.

physics.chem-ph↗

arXiv2024

Global Human-guided Counterfactual Explanations for Molecular Properties via Reinforcement Learning

Counterfactual explanations of Graph Neural Networks (GNNs) offer a powerful way to understand data that can naturally be represented by a graph structure. Furthermore, in many domains, it is highly desirable to derive data-driven global explanations or rules that can better explain the high-level properties of the models and data in question. However, evaluating global counterfactual explanations is hard in real-world datasets due to a lack of human-annotated ground truth, which limits their use in areas like molecular sciences. Additionally, the increasing scale of these datasets provides a challenge for random search-based methods. In this paper, we develop a novel global explanation model RLHEX for molecular property prediction. It aligns the counterfactual explanations with human-defined principles, making the explanations more interpretable and easy for experts to evaluate. RLHEX includes a VAE-based graph generator to generate global explanations and an adapter to adjust the latent representation space to human-defined principles. Optimized by Proximal Policy Optimization (PPO), the global explanations produced by RLHEX cover 4.12% more input graphs and reduce the distance between the counterfactual explanation set and the input set by 0.47% on average across three molecular datasets. RLHEX provides a flexible framework to incorporate different human-designed principles into the counterfactual explanation generation process, aligning these explanations with domain expertise. The code and data are released at https://github.com/dqwang122/RLHEX.

cs.LG↗

arXiv2024

TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation

Video generation has many unique challenges beyond those of image generation. The temporal dimension introduces extensive possible variations across frames, over which consistency and continuity may be violated. In this study, we move beyond evaluating simple actions and argue that generated videos should incorporate the emergence of new concepts and their relation transitions like in real-world videos as time progresses. To assess the Temporal Compositionality of video generation models, we propose TC-Bench, a benchmark of meticulously crafted text prompts, corresponding ground truth videos, and robust evaluation metrics. The prompts articulate the initial and final states of scenes, effectively reducing ambiguities for frame development and simplifying the assessment of transition completion. In addition, by collecting aligned real-world videos corresponding to the prompts, we expand TC-Bench's applicability from text-conditional models to image-conditional ones that can perform generative frame interpolation. We also develop new metrics to measure the completeness of component transitions in generated videos, which demonstrate significantly higher correlations with human judgments than existing metrics. Our comprehensive experimental results reveal that most video generators achieve less than 20% of the compositional changes, highlighting enormous space for future improvement. Our analysis indicates that current video generation models struggle to interpret descriptions of compositional changes and synthesize various components across different time steps.

cs.CV↗

arXiv2024

Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning

Prompt-based learning has been demonstrated as a compelling paradigm contributing to large language models' tremendous success (LLMs). Inspired by their success in language tasks, existing research has leveraged LLMs in embodied instruction following and task planning. In this work, we tackle the problem of training a robot to understand multimodal prompts, interleaving vision signals with text descriptions. This type of task poses a major challenge to robots' capability to understand the interconnection and complementarity between vision and language signals. In this work, we introduce an effective framework that learns a policy to perform robot manipulation with multimodal prompts from multi-task expert trajectories. Our methods consist of a two-stage training pipeline that performs inverse dynamics pretraining and multi-task finetuning. To facilitate multimodal understanding, we design our multimodal prompt encoder by augmenting a pretrained LM with a residual connection to the visual input and model the dependencies among action dimensions. Empirically, we evaluate the efficacy of our method on the VIMA-BENCH and establish a new state-of-the-art (10% improvement in success rate). Moreover, we demonstrate that our model exhibits remarkable in-context learning ability. Project page: \url{https://midas-icml.github.io/}.

cs.RO↗

arXiv2024

Cross-Task Defense: Instruction-Tuning LLMs for Content Safety

Recent studies reveal that Large Language Models (LLMs) face challenges in balancing safety with utility, particularly when processing long texts for NLP tasks like summarization and translation. Despite defenses against malicious short questions, the ability of LLMs to safely handle dangerous long content, such as manuals teaching illicit activities, remains unclear. Our work aims to develop robust defenses for LLMs in processing malicious documents alongside benign NLP task queries. We introduce a defense dataset comprised of safety-related examples and propose single-task and mixed-task losses for instruction tuning. Our empirical results demonstrate that LLMs can significantly enhance their capacity to safely manage dangerous content with appropriate instruction tuning. Additionally, strengthening the defenses of tasks most susceptible to misuse is effective in protecting LLMs against processing harmful information. We also observe that trade-offs between utility and safety exist in defense strategies, where Llama2, utilizing our proposed approach, displays a significantly better balance compared to Llama1.

cs.CL↗

arXiv2024

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo. CuMo incorporates Co-upcycled Top-K sparsely-gated Mixture-of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with minimal additional activated parameters during inference. CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruction tuning stage. Auxiliary losses are used to ensure a balanced loading of experts. CuMo outperforms state-of-the-art multimodal LLMs across various VQA and visual-instruction-following benchmarks using models within each model size group, all while training exclusively on open-sourced datasets. The code and model weights for CuMo are open-sourced at https://github.com/SHI-Labs/CuMo.

cs.CV↗

arXiv2024

Spectral form factor in chaotic, localized, and integrable open quantum many-body systems

We numerically study the spectral statistics of open quantum many-body systems (OQMBS) as signatures of quantum chaos (or the lack thereof), using the dissipative spectral form factor (DSFF), a generalization of the spectral form factor to complex spectra. We show that the DSFF of chaotic OQMBS generically displays the $\textit{quadratic}$ ramp-plateau behaviour of the Ginibre ensemble from random matrix theory, in contrast to the linear ramp-plateau behaviour of the Gaussian ensemble in closed quantum systems. Furthermore, in the presence of many-body interactions, such RMT behaviour emerges only after a time scale $τ_{\mathrm{dev}}$, which generally increases with system size for sufficiently large system size, and can be identified as the non-Hermitian analogue of the $\textit{many-body Thouless time}$. The universality of the random matrix theory behavior is demonstrated by surveying twelve models of OQMBS, including random Kraus circuits (quantum channels) and random Lindbladians (Liouvillians) in several symmetry classes, as well as Lindbladians of paradigmatic models such as the Sachdev-Ye-Kitaev (SYK), XXZ, and the transverse field Ising models. We devise an unfolding and filtering procedure to remove variations of the averaged density of states which would otherwise hide the universal RMT-like signatures in the DSFF for chaotic OQMBS. Beyond chaotic OQMBS, we study the spectral statistics of non-chaotic OQMBS, specifically the integrable XX model and a system in the many-body localized (MBL) regime in the presence of dissipation, which exhibit DSFF behaviours distinct from the ramp-plateau behaviour of random matrix theory. Lastly, we study the DSFF of Lindbladians with the Hamiltonian term set to zero, i.e. only the jump operators are present, and demonstrate that the results of RMT universality and scaling of many-body Thouless time survive even without coherent evolution.

cond-mat.stat-mech↗

arXiv2024

Chemical Potentials and the One-Electron Hamiltonian of the Second-Order Perturbation Theory from the Functional Derivative Approach

We develop a functional derivative approach to calculate the chemical potentials of the second-order perturbation theory (MP2). In the functional derivative approach, the correlation part of the MP2 chemical potential, which is the derivative of the MP2 correlation energy with respect to the occupation number of frontier orbitals, is obtained from the chain rule via the non-interacting Green's function. First, the MP2 correlation energy is expressed in terms of the non-interacting Green's function and its functional derivative to the non-interacting Green's function is the second-order self-energy. Then the derivative of the non-interacting Green's function to the occupation number is obtained by including the orbital relaxation effect. We show that the MP2 chemical potentials obtained from the functional derivative approach agrees with that obtained from the finite difference approach. The one-electron Hamiltonian, defined as the derivative of the MP2 energy with respect to the one particle density matrix, is also derived using the functional derivative approach, which can be used in the self-consistent calculations of MP2 and double-hybrid density functionals. The developed functional derivative approach is promising for calculating the chemical potentials and the one-electron Hamiltonian of approximate functionals and many-body perturbation approaches dependent explicitly on the non-interacting Green's function.

physics.chem-ph↗

arXiv2024

"I Wish There Were an AI": Challenges and AI Potential in Cancer Patient-Provider Communication

Patient-provider communication has been crucial to cancer patients' survival after their cancer treatments. However, the research community and patients themselves often overlook the communication challenges after cancer treatments as they are overshadowed by the severity of the patient's illness and the variety and rarity of the cancer disease itself. Meanwhile, the recent technical advances in AI, especially in Large Language Models (LLMs) with versatile natural language interpretation and generation ability, demonstrate great potential to support communication in complex real-world medical situations. By interviewing six healthcare providers and eight cancer patients, our goal is to explore the providers' and patients' communication barriers in the post-cancer treatment recovery period, their expectations for future communication technologies, and the potential of AI technologies in this context. Our findings reveal several challenges in current patient-provider communication, including the knowledge and timing gaps between cancer patients and providers, their collaboration obstacles, and resource limitations. Moreover, based on providers' and patients' needs and expectations, we summarize a set of design implications for intelligent communication systems, especially with the power of LLMs. Our work sheds light on the design of future AI-powered systems for patient-provider communication under high-stake and high-uncertainty situations.

cs.HC↗

arXiv2024

Predicting Future Spatiotemporal Occupancy Grids with Semantics for Autonomous Driving

For autonomous vehicles to proactively plan safe trajectories and make informed decisions, they must be able to predict the future occupancy states of the local environment. However, common issues with occupancy prediction include predictions where moving objects vanish or become blurred, particularly at longer time horizons. We propose an environment prediction framework that incorporates environment semantics for future occupancy prediction. Our method first semantically segments the environment and uses this information along with the occupancy information to predict the spatiotemporal evolution of the environment. We validate our approach on the real-world Waymo Open Dataset. Compared to baseline methods, our model has higher prediction accuracy and is capable of maintaining moving object appearances in the predictions for longer prediction time horizons.

cs.RO↗

arXiv2024

More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering

While most existing works on LLM prompting techniques focus only on how to select a better set of data samples inside one single prompt input (In-Context Learning or ICL), why can not we design and leverage multiple prompts together to further improve the LLM's performance? In this work, we propose In-Context Sampling (ICS), a low-resource LLM prompting technique to produce confident predictions by optimizing the construction of multiple ICL prompt inputs. Extensive experiments with three open-source LLMs (FlanT5-XL, Mistral-7B, and Mixtral-8x7B) on four NLI datasets (e-SNLI, Multi-NLI, ANLI, and Contract-NLI) and one QA dataset (CommonsenseQA) illustrate that ICS can consistently enhance LLMs' performance. An in-depth evaluation with three data similarity-based ICS strategies suggests that these strategies can further elevate LLM's performance, which sheds light on a new yet promising future research direction.

cs.CL↗