Searcharxiv⌕ Search

arXiv subjects

Chen Liang

Publications and source records attributed to Chen Liang.

At least 127 records · Page 7Linked to original sources
arXiv2023

A Multi-day Needs-based Modeling Approach for Activity and Travel Demand Analysis

This paper proposes a multi-day needs-based model for activity and travel demand analysis. The model captures the multi-day dynamics in activity generation, which enables the modeling of activities with increased flexibility in time and space (e.g., e-commerce and remote working). As an enhancement to activity-based models, the proposed model captures the underlying decision-making process of activity generation by accounting for psychological needs as the drivers of activities. The level of need satisfaction is modeled as a psychological inventory, whose utility is optimized via decisions on activity participation, location, and duration. The utility includes both the benefit in the inventory gained and the cost in time, monetary expense as well as maintenance of safety stock. The model includes two sub-models, a Deterministic Model that optimizes the utility of the inventory, and an Empirical Model that accounts for heterogeneity and stochasticity. Numerical experiments are conducted to demonstrate model scalability. A maximum likelihood estimator is proposed, the properties of the log-likelihood function are examined and the recovery of true parameters is tested. This research contributes to the literature on transportation demand models in the following three aspects. First, it is arguably better grounded in psychological theory than traditional models and allows the generation of activity patterns to be policy-sensitive (while avoiding the need for ad hoc utility definitions). Second, it contributes to the development of needs-based models with a non-myopic approach to model multi-day activity patterns. Third, it proposes a tractable model formulation via problem reformulation and computational enhancements, which allows for maximum likelihood parameter estimation.

eess.SY↗
arXiv2023

TARGET: Template-Transferable Backdoor Attack Against Prompt-based NLP Models via GPT4

Prompt-based learning has been widely applied in many low-resource NLP tasks such as few-shot scenarios. However, this paradigm has been shown to be vulnerable to backdoor attacks. Most of the existing attack methods focus on inserting manually predefined templates as triggers in the pre-training phase to train the victim model and utilize the same triggers in the downstream task to perform inference, which tends to ignore the transferability and stealthiness of the templates. In this work, we propose a novel approach of TARGET (Template-trAnsfeRable backdoor attack aGainst prompt-basEd NLP models via GPT4), which is a data-independent attack method. Specifically, we first utilize GPT4 to reformulate manual templates to generate tone-strong and normal templates, and the former are injected into the model as a backdoor trigger in the pre-training phase. Then, we not only directly employ the above templates in the downstream task, but also use GPT4 to generate templates with similar tone to the above templates to carry out transferable attacks. Finally we have conducted extensive experiments on five NLP datasets and three BERT series models, with experimental results justifying that our TARGET method has better attack performance and stealthiness compared to the two-external baseline methods on direct attacks, and in addition achieves satisfactory attack capability in the unseen tone-similar templates.

cs.CL↗
arXiv2023

LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models

Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning. In this work we focus on the scenario where quantization and LoRA fine-tuning are applied together on a pre-trained model. In such cases it is common to observe a consistent gap in the performance on downstream tasks between full fine-tuning and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that simultaneously quantizes an LLM and finds a proper low-rank initialization for LoRA fine-tuning. Such an initialization alleviates the discrepancy between the quantized and full-precision model and significantly improves generalization in downstream tasks. We evaluate our method on natural language understanding, question answering, summarization, and natural language generation tasks. Experiments show that our method is highly effective and outperforms existing quantization methods, especially in the challenging 2-bit and 2/4-bit mixed precision regimes. The code is available on https://github.com/yxli2123/LoftQ.

cs.CL↗
arXiv2023

Provably Robust Semi-Infinite Program Under Collision Constraints via Subdivision

We present a semi-infinite program (SIP) solver for trajectory optimizations of general articulated robots. These problems are more challenging than standard Nonlinear Program (NLP) by involving an infinite number of non-convex, collision constraints. Prior SIP solvers based on constraint sampling cannot guarantee the satisfaction of all constraints. Instead, our method uses a conservative bound on articulated body motions to ensure the solution feasibility throughout the optimization procedure. We further use subdivision to adaptively reduce the error in conservative motion estimation. Combined, we prove that our SIP solver guarantees feasibility while approaches the critical point of SIP problems up to arbitrary user-provided precision. We have verified our method on a row of trajectory optimization problems involving industrial robot arms and UAVs, where our method can generate collision-free, locally optimal trajectories within a couple minutes.

cs.RO↗
arXiv2023

Second-Order Convergent Collision-Constrained Optimization-Based Planner

Finding robot poses and trajectories represents a foundational aspect of robot motion planning. Despite decades of research, efficiently and robustly addressing these challenges is still difficult. Existing approaches are often plagued by various limitations, such as intricate geometric approximations, violations of collision constraints, or slow first-order convergence. In this paper, we introduce two novel optimization formulations that offer provable robustness, achieving second-order convergence while requiring only a convex approximation of the robot's links and obstacles. Our first method, known as the Explicit Collision Barrier (ECB) method, employs a barrier function to guarantee separation between convex objects. ECB uses an efficient matrix factorization technique, enabling a second-order Newton's method with an iterative complexity linear in the number of separating planes. Our second method, referred to as the Implicit Collision Barrier (ICB) method, further transforms the separating planes into implicit functions of robot poses. We show such an implicit objective function is twice-differentiable, with derivatives evaluated at a linear complexity. To assess the effectiveness of our approaches, we conduct a comparative study with a first-order baseline algorithm across six testing scenarios. Our results unequivocally justify that our method exhibits significantly faster convergence rates compared to the baseline algorithm.

cs.RO↗
arXiv2023

Learning Reduced-Order Soft Robot Controller

Deformable robots are notoriously difficult to model or control due to its high-dimensional configuration spaces. Direct trajectory optimization suffers from the curse-of-dimensionality and incurs a high computational cost, while learning-based controller optimization methods are sensitive to hyper-parameter tuning. To overcome these limitations, we hypothesize that high fidelity soft robots can be both simulated and controlled by restricting to low-dimensional spaces. Under such assumption, we propose a two-stage algorithm to identify such simulation- and control-spaces. Our method first identifies the so-called simulation-space that captures the salient deformation modes, to which the robot's governing equation is restricted. We then identify the control-space, to which control signals are restricted. We propose a multi-fidelity Riemannian Bayesian bilevel optimization to identify task-specific control spaces. We show that the dimension of control-space can be less than $10$ for a high-DOF soft robot to accomplish walking and swimming tasks, allowing low-dimensional MPC controllers to be applied to soft robots with tractable computational complexity.

cs.RO↗
arXiv2023

Rethinking the Cloudonomics of Efficient I/O for Data-Intensive Analytics Applications

This paper explores a prevailing trend in the industry: migrating data-intensive analytics applications from on-premises to cloud-native environments. We find that the unique cost models associated with cloud-based storage necessitate a more nuanced understanding of optimizing performance. Specifically, based on traces collected from Uber's Presto fleet in production, we argue that common I/O optimizations, such as table scan and filter, and broadcast join, may lead to unexpected costs when naively applied in the cloud. This is because traditional I/O optimizations mainly focus on improving throughput or latency in on-premises settings, without taking into account the monetary costs associated with storage API calls. In cloud environments, these costs can be significant, potentially involving billions of API calls per day just for Presto workloads at Uber scale. Presented as a case study, this paper serves as a starting point for further research to design efficient I/O strategies specifically tailored for data-intensive applications in cloud settings.

cs.DC↗
arXiv2023

Module-wise Adaptive Distillation for Multimodality Foundation Models

Pre-trained multimodal foundation models have demonstrated remarkable generalizability but pose challenges for deployment due to their large sizes. One effective approach to reducing their sizes is layerwise distillation, wherein small student models are trained to match the hidden representations of large teacher models at each layer. Motivated by our observation that certain architecture components, referred to as modules, contribute more significantly to the student's performance than others, we propose to track the contributions of individual modules by recording the loss decrement after distillation each module and choose the module with a greater contribution to distill more frequently. Such an approach can be naturally formulated as a multi-armed bandit (MAB) problem, where modules and loss decrements are considered as arms and rewards, respectively. We then develop a modified-Thompson sampling algorithm named OPTIMA to address the nonstationarity of module contributions resulting from model updating. Specifically, we leverage the observed contributions in recent history to estimate the changing contribution of each module and select modules based on these estimations to maximize the cumulative contribution. We evaluate the effectiveness of OPTIMA through distillation experiments on various multimodal understanding and image captioning tasks, using the CoCa-Large model (Yu et al., 2022) as the teacher model.

cs.CV↗
arXiv2023

Excited Dirac stars with higher azimuthal harmonic index

In this paper, we investigate the properties of the first excited state Dirac stars (DSs) with higher azimuthal harmonic index (specifically, the azimuthal harmonic indexes $m_D$ = $3/2$, $5/2$, $7/2$), as well as the relationship between the ADM mass and angular momentum of Dirac stars with respect to frequency. Moreover, We find that the ergospheres of DSs appear at lower spinor field frequencies, and both the ergospheres and the distribution of the spinor field functions are asymmetric about the equatorial plane. Furthermore, we introduce the ground state scalar field and examine its impact on this system, which is known as the multi-state Dirac-boson stars (DBSs) model. We show various types of solution families for DBSs under both synchronized frequency $ω$ and nonsynchronized frequencies and find that similar to DSs, the spinor field and the ergospheres of DBSs are also asymmetric about the equatorial plane, but the ergospheres appear at higher spinor field frequencies.

gr-qc↗
arXiv2023

Dirac-Proca stars

We consider a model consists of the Einstein gravity in four-dimensional spacetime, a Proca field and two Dirac fields through minimum coupling. By numerically solving this model, we obtain two types of solutions: synchronized frequency solutions and non-synchronized frequency solutions. We find that in the case of two kinds of matter fields, different families of solutions can be obtained in both synchronized and non-synchronized frequency cases, and the two families can shift to each other when certain parameters are changed. Moreover, we calculate the binding energy of multi-field solutions and give the relationship between binding energy $E$ and synchronized frequency $\tildeω$ (non-synchronized frequency $\tildeω_P$), so as to analyze the stability of the corresponding family.

gr-qc↗
arXiv2023

Exploring the Opportunities of AR for Enriching Storytelling with Family Photos between Grandparents and Grandchildren

Storytelling with family photos, as an important mode of reminiscence-based activities, can be instrumental in promoting intergenerational communication between grandparents and grandchildren by strengthening generation bonds and shared family values. Motivated by challenges that existing technology approaches encountered for improving intergenerational storytelling (e.g., the need to hold the tablet, the potential view detachment from the physical world in Virtual Reality (VR)), we sought to find new ways of using Augmented Reality (AR) to support intergenerational storytelling, which offers new capabilities (e.g., 3D models, new interactivity) to enhance the expression for the storyteller. We conducted a two-part exploratory study, where pairs of grandparents and grandchildren 1) participated in an in-person storytelling activity with a semi-structured interview 2) and then a participatory design session with AR technology probes that we designed to inspire their exploration. Our findings revealed insights into the possible ways of intergenerational storytelling, the feasibility and usages of AR in facilitating it, and the key design implications for leveraging AR in intergenerational storytelling.

cs.HC↗
arXiv2023

Logic-induced Diagnostic Reasoning for Semi-supervised Semantic Segmentation

Recent advances in semi-supervised semantic segmentation have been heavily reliant on pseudo labeling to compensate for limited labeled data, disregarding the valuable relational knowledge among semantic concepts. To bridge this gap, we devise LogicDiag, a brand new neural-logic semi-supervised learning framework. Our key insight is that conflicts within pseudo labels, identified through symbolic knowledge, can serve as strong yet commonly ignored learning signals. LogicDiag resolves such conflicts via reasoning with logic-induced diagnoses, enabling the recovery of (potentially) erroneous pseudo labels, ultimately alleviating the notorious error accumulation problem. We showcase the practical application of LogicDiag in the data-hungry segmentation scenario, where we formalize the structured abstraction of semantic concepts as a set of logic rules. Extensive experiments on three standard semi-supervised semantic segmentation benchmarks demonstrate the effectiveness and generality of LogicDiag. Moreover, LogicDiag highlights the promising opportunities arising from the systematic integration of symbolic reasoning into the prevalent statistical, neural learning approaches.

cs.CV↗
arXiv2023

Towards Ubiquitous Intelligent Hand Interaction

The development of ubiquitous computing and sensing devices has brought about novel interaction scenarios such as mixed reality and IoT (e.g., smart home), which pose new demands for the next generation of natural user interfaces (NUI). Human hand, benefit for the large degree-of-freedom, serves as a medium through which people interact with the external world in their daily lives, thus also being regarded as the main entry of NUI. Unfortunately, current hand tracking system is largely confined on first perspective vision-based solutions, which suffer from optical artifacts and are not practical in ubiquitous environments. In my thesis, I rethink this problem by analyzing the underlying logic in terms of sensor, behavior, and semantics, constituting a research framework for achieving ubiquitous intelligent hand interaction. Then I summarize my previous research topics and illustrated the future research directions based on my research framework.

cs.HC↗
arXiv2023

LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

Transformer models have achieved remarkable results in various natural language tasks, but they are often prohibitively large, requiring massive memories and computational resources. To reduce the size and complexity of these models, we propose LoSparse (Low-Rank and Sparse approximation), a novel model compression technique that approximates a weight matrix by the sum of a low-rank matrix and a sparse matrix. Our method combines the advantages of both low-rank approximations and pruning, while avoiding their limitations. Low-rank approximation compresses the coherent and expressive parts in neurons, while pruning removes the incoherent and non-expressive parts in neurons. Pruning enhances the diversity of low-rank approximations, and low-rank approximation prevents pruning from losing too many expressive neurons. We evaluate our method on natural language understanding, question answering, and natural language generation tasks. We show that it significantly outperforms existing compression methods.

cs.LG↗
arXiv2023

Multi-state Dirac stars

In this paper, we construct the multi-state Dirac stars (MSDSs) consisting of two pairs of Dirac fields. The two pairs of Dirac fields are in the ground state and the first excited state, respectively. Each pair consists of two fields with opposite spins, ensuring spherical symmetry of the system. We discuss the solutions of the MSDSs under synchronized and nonsynchronized frequencies. By varying the mass $\tildeμ_1$ of the excited state Dirac field and the frequency $\tildeω_0$ of the ground state Dirac field, we obtain different types of solutions, including single-branch and double-branch solutions. These two types of solutions do not smoothly transition into each other as the parameters $\tildeμ_1$ and $\tildeω_0$ continuously change, but undergo a sudden transition when $\tildeμ_1$ ($\tildeω_0$) is greater than or less than the threshold value of $0.7694$ ($0.733$). Furthermore, we analyze the characteristics of the various MSDSs solutions and analyze the relationship between the ADM mass $M$ of the MSDSs and the synchronized and nonsynchronized frequencies. Subsequently, we calculate the binding energy $E_B$ of the MSDSs and discuss the stability of the solutions. Finally, we discuss the feasibility of simulating the dark matter halos using MSDSs.

hep-th↗
arXiv2023

Less is More: Task-aware Layer-wise Distillation for Language Model Compression

Layer-wise distillation is a powerful tool to compress large models (i.e. teacher models) into small ones (i.e., student models). The student distills knowledge from the teacher by mimicking the hidden representations of the teacher at every intermediate layer. However, layer-wise distillation is difficult. Since the student has a smaller model capacity than the teacher, it is often under-fitted. Furthermore, the hidden representations of the teacher contain redundant information that the student does not necessarily need for the target task's learning. To address these challenges, we propose a novel Task-aware layEr-wise Distillation (TED). TED designs task-aware filters to align the hidden representations of the student and the teacher at each layer. The filters select the knowledge that is useful for the target task from the hidden representations. As such, TED reduces the knowledge gap between the two models and helps the student to fit better on the target task. We evaluate TED in two scenarios: continual pre-training and fine-tuning. TED demonstrates significant and consistent improvements over existing distillation methods in both scenarios. Code is available at https://github.com/cliang1453/task-aware-distillation.

cs.CL↗
arXiv2023

Symbolic Discovery of Optimization Algorithms

We present a method to formulate algorithm discovery as program search, and apply it to discover optimization algorithms for deep neural network training. We leverage efficient search techniques to explore an infinite and sparse program space. To bridge the large generalization gap between proxy and target tasks, we also introduce program selection and simplification strategies. Our method discovers a simple and effective optimization algorithm, $\textbf{Lion}$ ($\textit{Evo$\textbf{L}$ved S$\textbf{i}$gn M$\textbf{o}$me$\textbf{n}$tum}$). It is more memory-efficient than Adam as it only keeps track of the momentum. Different from adaptive optimizers, its update has the same magnitude for each parameter calculated through the sign operation. We compare Lion with widely used optimizers, such as Adam and Adafactor, for training a variety of models on different tasks. On image classification, Lion boosts the accuracy of ViT by up to 2% on ImageNet and saves up to 5x the pre-training compute on JFT. On vision-language contrastive learning, we achieve 88.3% $\textit{zero-shot}$ and 91.1% $\textit{fine-tuning}$ accuracy on ImageNet, surpassing the previous best results by 2% and 0.1%, respectively. On diffusion models, Lion outperforms Adam by achieving a better FID score and reducing the training compute by up to 2.3x. For autoregressive, masked language modeling, and fine-tuning, Lion exhibits a similar or better performance compared to Adam. Our analysis of Lion reveals that its performance gain grows with the training batch size. It also requires a smaller learning rate than Adam due to the larger norm of the update produced by the sign function. Additionally, we examine the limitations of Lion and identify scenarios where its improvements are small or not statistically significant. Lion is also successfully deployed in production systems such as Google search ads CTR model.

cs.LG↗
arXiv2023

Optimization design of a micro-perforated panel absorber with 8.6 octave bands

In order to improve low-frequency characteristics of micro-perforated panel absorbers, sound absorption structures composed of micro-perforated panels and expansion chambers are design, and an optimization design method is constructed based on the transfer function model and the simulated annealing algorithm. First, a single-chamber structure composed of a micro-perforated panel and an expansion chamber is build, and the sound absorption curve is simulated by the finite element method. Second, for the sake of enlarging the continuous absorption bandwidth with absorption coefficients not less than 0.8, a three-chamber structure is designed, which has a sound absorption bandwidth of 1277Hz (27-1304Hz) covering 5.6 octave bands. Then, the transfer function model of the structure is established, and a series of theoretical formulae are derived to calculate the absorption coefficients. Subsequently, the sound absorption bandwidths calculated by the theoretical formulae and the finite element method are compared, and the relative error is 3.68%. Finally, an optimization design method is constructed by combining the transfer function model and the simulated annealing algorithm, where the optimization objective is to maximize the absorption bandwidth and the optimization variables are structural parameters of the three-chamber structure. The results show, after optimization, the three-chamber structure exhibits an excellent sound absorption performance, with a continuous bandwidth of 1591Hz (4-1595Hz), realizing 8.6 octave bands.

eess.AS↗