arXiv subjects

Han Zhao

Publications and source records attributed to Han Zhao.

At least 181 records · Page 10Linked to original sources

arXiv2023

Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning

Contrastive loss has been increasingly used in learning representations from multiple modalities. In the limit, the nature of the contrastive loss encourages modalities to exactly match each other in the latent space. Yet it remains an open question how the modality alignment affects the downstream task performance. In this paper, based on an information-theoretic argument, we first prove that exact modality alignment is sub-optimal in general for downstream prediction tasks. Hence we advocate that the key of better performance lies in meaningful latent modality structures instead of perfect modality alignment. To this end, we propose three general approaches to construct latent modality structures. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization. Extensive experiments are conducted on two popular multi-modal representation learning frameworks: the CLIP-based two-tower model and the ALBEF-based fusion model. We test our model on a variety of tasks including zero/few-shot image classification, image-text retrieval, visual question answering, visual reasoning, and visual entailment. Our method achieves consistent improvements over existing methods, demonstrating the effectiveness and generalizability of our proposed approach on latent modality structure regularization.

cs.LG↗

arXiv2023

Dual-View Selective Instance Segmentation Network for Unstained Live Adherent Cells in Differential Interference Contrast Images

Despite recent advances in data-independent and deep-learning algorithms, unstained live adherent cell instance segmentation remains a long-standing challenge in cell image processing. Adherent cells' inherent visual characteristics, such as low contrast structures, fading edges, and irregular morphology, have made it difficult to distinguish from one another, even by human experts, let alone computational methods. In this study, we developed a novel deep-learning algorithm called dual-view selective instance segmentation network (DVSISN) for segmenting unstained adherent cells in differential interference contrast (DIC) images. First, we used a dual-view segmentation (DVS) method with pairs of original and rotated images to predict the bounding box and its corresponding mask for each cell instance. Second, we used a mask selection (MS) method to filter the cell instances predicted by the DVS to keep masks closest to the ground truth only. The developed algorithm was trained and validated on our dataset containing 520 images and 12198 cells. Experimental results demonstrate that our algorithm achieves an AP_segm of 0.555, which remarkably overtakes a benchmark by a margin of 23.6%. This study's success opens up a new possibility of using rotated images as input for better prediction in cell images.

cs.CV↗

arXiv2022

Fundamental Limits and Tradeoffs in Invariant Representation Learning

A wide range of machine learning applications such as privacy-preserving learning, algorithmic fairness, and domain adaptation/generalization among others, involve learning invariant representations of the data that aim to achieve two competing goals: (a) maximize information or accuracy with respect to a target response, and (b) maximize invariance or independence with respect to a set of protected features (e.g., for fairness, privacy, etc). Despite their wide applicability, theoretical understanding of the optimal tradeoffs -- with respect to accuracy, and invariance -- achievable by invariant representations is still severely lacking. In this paper, we provide an information theoretic analysis of such tradeoffs under both classification and regression settings. More precisely, we provide a geometric characterization of the accuracy and invariance achievable by any representation of the data; we term this feasible region the information plane. We provide an inner bound for this feasible region for the classification case, and an exact characterization for the regression case, which allows us to either bound or exactly characterize the Pareto optimal frontier between accuracy and invariance. Although our contributions are mainly theoretical, a key practical application of our results is in certifying the potential sub-optimality of any given representation learning algorithm for either classification or regression tasks. Our results shed new light on the fundamental interplay between accuracy and invariance, and may be useful in guiding the design of future representation learning algorithms.

cs.LG↗

arXiv2022

Algorithms and Theory for Supervised Gradual Domain Adaptation

The phenomenon of data distribution evolving over time has been observed in a range of applications, calling the needs of adaptive learning algorithms. We thus study the problem of supervised gradual domain adaptation, where labeled data from shifting distributions are available to the learner along the trajectory, and we aim to learn a classifier on a target data distribution of interest. Under this setting, we provide the first generalization upper bound on the learning error under mild assumptions. Our results are algorithm agnostic, general for a range of loss functions, and only depend linearly on the averaged learning error across the trajectory. This shows significant improvement compared to the previous upper bound for unsupervised gradual domain adaptation, where the learning error on the target domain depends exponentially on the initial error on the source domain. Compared with the offline setting of learning from multiple domains, our results also suggest the potential benefits of the temporal structure among different domains in adapting to the target one. Empirically, our theoretical results imply that learning proper representations across the domains will effectively mitigate the learning errors. Motivated by these theoretical insights, we propose a min-max learning objective to learn the representation and classifier simultaneously. Experimental results on both semi-synthetic and large-scale real datasets corroborate our findings and demonstrate the effectiveness of our objectives.

cs.LG↗

arXiv2022

Conditional Supervised Contrastive Learning for Fair Text Classification

Contrastive representation learning has gained much attention due to its superior performance in learning representations from both image and sequential data. However, the learned representations could potentially lead to performance disparities in downstream tasks, such as increased silencing of underrepresented groups in toxicity comment classification. In light of this challenge, in this work, we study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives, and then propose to use conditional supervised contrastive objectives to learn fair representations for text classification. We conduct experiments on two text datasets to demonstrate the effectiveness of our approaches in balancing the trade-offs between task performance and bias mitigation among existing baselines for text classification. Furthermore, we also show that the proposed methods are stable in different hyperparameter settings.

cs.CL↗

arXiv2022

Electro-optic transduction in silicon via GHz-frequency nanomechanics

Interfacing electronics with optical fiber networks is key to the long-distance transfer of classical and quantum information. Piezo-optomechanical transducers enable such interfaces by using GHz-frequency acoustic vibrations as mediators for converting microwave photons to optical photons via the combination of optomechanical and piezoelectric interactions. However, despite successful demonstrations, efficient piezo-optomechanical transduction remains out of reach due to the challenges associated with hybrid material integration and increased loss from piezoelectric materials when operating in the quantum regime. Here, we demonstrate an alternative approach in which we actuate 5-GHz phonons in a conventional silicon-on-insulator platform. In our experiment, microwave photons resonantly drive a phononic crystal oscillator via the electrostatic force realized in a charge-biased narrow-gap capacitor. The mechanical vibrations are subsequently transferred via a phonon waveguide to an optomechanical cavity, where they transform into optical photons in the sideband of a pump laser field. Operating at room temperature and atmospheric pressure, we measure a microwave-to-optical photon conversion efficiency of $1.8 \times 10^{-7}$ in a 3.3 MHz bandwidth, and demonstrate efficient phase modulation with a half-wave voltage of $V_π= 750 $ mV. Our results mark a stepping stone towards quantum transduction with integrated devices made from crystalline silicon, which promise efficient high-bandwidth operation, and integration with superconducting qubits. Additionally, the lack of need for piezoelectricity or other intrinsic nonlinearities makes our approach adaptable to a wide range of materials for potential applications beyond quantum technologies.

physics.optics↗

arXiv2022

Greedy Modality Selection via Approximate Submodular Maximization

Multimodal learning considers learning from multi-modality data, aiming to fuse heterogeneous sources of information. However, it is not always feasible to leverage all available modalities due to memory constraints. Further, training on all the modalities may be inefficient when redundant information exists within data, such as different subsets of modalities providing similar performance. In light of these challenges, we study modality selection, intending to efficiently select the most informative and complementary modalities under certain computational constraints. We formulate a theoretical framework for optimizing modality selection in multimodal learning and introduce a utility measure to quantify the benefit of selecting a modality. For this optimization problem, we present efficient algorithms when the utility measure exhibits monotonicity and approximate submodularity. We also connect the utility measure with existing Shapley-value-based feature importance scores. Last, we demonstrate the efficacy of our algorithm on synthetic (Patch-MNIST) and two real-world (PEMS-SF, CMU-MOSI) datasets.

cs.LG↗

arXiv2022

Dense-TNT: Efficient Vehicle Type Classification Neural Network Using Satellite Imagery

Accurate vehicle type classification serves a significant role in the intelligent transportation system. It is critical for ruler to understand the road conditions and usually contributive for the traffic light control system to response correspondingly to alleviate traffic congestion. New technologies and comprehensive data sources, such as aerial photos and remote sensing data, provide richer and high-dimensional information. Also, due to the rapid development of deep neural network technology, image based vehicle classification methods can better extract underlying objective features when processing data. Recently, several deep learning models have been proposed to solve the problem. However, traditional pure convolutional based approaches have constraints on global information extraction, and the complex environment, such as bad weather, seriously limits the recognition capability. To improve the vehicle type classification capability under complex environment, this study proposes a novel Densely Connected Convolutional Transformer in Transformer Neural Network (Dense-TNT) framework for the vehicle type classification by stacking Densely Connected Convolutional Network (DenseNet) and Transformer in Transformer (TNT) layers. Three-region vehicle data and four different weather conditions are deployed for recognition capability evaluation. Experimental findings validate the recognition ability of our proposed vehicle classification model with little decay, even under the heavy foggy weather condition.

cs.CV↗

arXiv2022

Interpolation-based immersed finite element and isogeometric analysis

We introduce a new paradigm for immersed finite element and isogeometric methods based on interpolating function spaces from an unfitted background mesh into Lagrange finite element spaces defined on a foreground mesh that captures the domain geometry but is otherwise subject to minimal constraints on element quality or connectivity. This is a generalization of the concept of Lagrange extraction from the isogeometric analysis literature and also related to certain variants of the finite cell and material point methods. Crucially, the interpolation may be approximate without sacrificing high-order convergence rates, which distinguishes the present method from existing finite cell, CutFEM, and immersogeometric approaches. The interpolation paradigm also permits non-invasive reuse of existing finite element software for immersed analysis. We analyze the properties of the interpolation-based immersed paradigm for a model problem and implement it on top of the open-source FEniCS finite element software, to apply it to a variety of problems in fluid, solid, and structural mechanics where we demonstrate high-order accuracy and applicability to practical geometries like trimmed spline patches.

math.NA↗

arXiv2022

A SUMO Framework for Deep Reinforcement Learning Experiments Solving Electric Vehicle Charging Dispatching Problem

In modern cities, the number of Electric vehicles (EV) is increasing rapidly for their low emission and better dynamic performance, leading to increasing demand for EV charging. However, due to the limited number of EV charging facilities, catering to the huge demand for time-consuming EV charging becomes a critical problem. It is quite a challenge to dispatch EVs in the dynamic traffic environment and coordinate interaction among agents. To better serve further research on various related Deep Reinforcment Learning (DRL) EV dispatching algorithms, an efficient simulation environment is necessary to ensure success. As simulator Simulation Urban Mobility (SUMO) is one of the most widely used open-source simulators, it has great significance in creating an environment that satisfies research requirements on SUMO. We aim to improve the efficiency of EV charging station usage and save time for EV users in further work. As a result, we design an EV navigation system on the basis of the traffic simulator SUMO using Jurong Area, Singapore in this paper. Various state-of-the-art DRL algorithms are deployed on the designed testbed to validate the feasibility of the framework in terms of EV charging dispatching problems. Besides EV dispatching problems, the environment can also serve for other reinforcement learning (RL) traffic control problems

eess.SY↗

arXiv2022

Exploring Gradient-based Multi-directional Controls in GANs

Generative Adversarial Networks (GANs) have been widely applied in modeling diverse image distributions. However, despite its impressive applications, the structure of the latent space in GANs largely remains as a black-box, leaving its controllable generation an open problem, especially when spurious correlations between different semantic attributes exist in the image distributions. To address this problem, previous methods typically learn linear directions or individual channels that control semantic attributes in the image space. However, they often suffer from imperfect disentanglement, or are unable to obtain multi-directional controls. In this work, in light of the above challenges, we propose a novel approach that discovers nonlinear controls, which enables multi-directional manipulation as well as effective disentanglement, based on gradient information in the learned GAN latent space. More specifically, we first learn interpolation directions by following the gradients from classification networks trained separately on the attributes, and then navigate the latent space by exclusively controlling channels activated for the target attribute in the learned directions. Empirically, with small training data, our approach is able to gain fine-grained controls over a diverse set of bi-directional and multi-directional attributes, and we showcase its ability to achieve disentanglement significantly better than state-of-the-art methods both qualitatively and quantitatively.

cs.CV↗

arXiv2022

Application of Data Encryption in Chinese Named Entity Recognition

Recently, with the continuous development of deep learning, the performance of named entity recognition tasks has been dramatically improved. However, the privacy and the confidentiality of data in some specific fields, such as biomedical and military, cause insufficient data to support the training of deep neural networks. In this paper, we propose an encryption learning framework to address the problems of data leakage and inconvenient disclosure of sensitive data in certain domains. We introduce multiple encryption algorithms to encrypt training data in the named entity recognition task for the first time. In other words, we train the deep neural network using the encrypted data. We conduct experiments on six Chinese datasets, three of which are constructed by ourselves. The experimental results show that the encryption method achieves satisfactory results. The performance of some models trained with encrypted data even exceeds the performance of the unencrypted method, which verifies the effectiveness of the introduced encryption method and solves the problem of data leakage to a certain extent.

cs.CR↗

arXiv2022

A quantum electromechanical interface for long-lived phonons

Controlling long-lived mechanical oscillators in the quantum regime holds promises for quantum information processing. Here, we present an electromechanical system capable of operating in the GHz-frequency band in a silicon-on-insulator platform. Relying on a novel driving scheme based on an electrostatic field and high-impedance microwave cavities based on TiN superinductors, we are able to demonstrate a parametrically-enhanced electromechanical coupling of ${g/2 π} = 1.1$ MHz, sufficient to enter the strong-coupling regime with a cooperativity of $\mathcal{C} = 1200$. The absence of piezoelectric materials in our platform leads to long mechanical lifetimes, finding intrinsic values up to $τ_\text{d} = 265~ μ$s ($Q = 8.4 \times {10}^6$ at $ω_\mathrm{m}/2π= 5$ GHz) measured at low-phonon numbers and millikelvin temperatures. Despite the strong parametric drives, we find the cavity-mechanics system in the quantum ground state by performing sideband thermometry measurements. Simultaneously achieving ground-state operation, long mechanical lifetimes, and strong coupling sets the stage for employing silicon electromechanical resonators as memory elements and transducers in hybrid quantum systems, and as a tool for probing the origins of acoustic loss in the quantum regime.

quant-ph↗

arXiv2022

Quantitative Versions of the Two-dimensional Gaussian Product Inequalities

The Gaussian product inequality (GPI) conjecture is one of the most famous inequalities associated with Gaussian distributions and has attracted a lot of concerns. In this note, we investigate the quantitative versions of the two-dimensional Gaussian product inequalities. For any centered non-degenerate two-dimensional Gaussian random vector $(X_1, X_2)$ with variances $σ_1^2, σ_2^2$ and the correlation coefficient $ρ$, we prove that for any real numbers $α_1, α_2\in (-1,0)$ or $α_1, α_2\in (0,\infty)$, it holds that %there exist functions of $α_1, α_2$ and $ρ$ such that $${\bf E}[|X_1|^{α_1}|X_2|^{α_2}]-{\bf E}[|X_1|^{α_1}]{\bf E}[|X_2|^{α_2}]\ge f(σ_1,σ_2,α_1, α_2, ρ)\ge 0, $$ where the function $f(σ_1,σ_2,α_1, α_2, ρ)$ will be given explicitly by Gamma function and is positive when $ρ\neq 0$. When $-1<α_1<0$ and $α_2>0,$ Russell and Sun (arXiv: 2205.10231v1) proved the "opposite Gaussian product inequality", of which we will also give a quantitative version. These quantitative inequalities are derived by employing the hypergeometric functions and the generalized hypergeometric functions.

math.PR↗

arXiv2022

Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning

Despite achieving state-of-the-art zero-shot performance, existing vision-language models still fall short of few-shot transfer ability on domain-specific problems. Classical fine-tuning often fails to prevent highly expressive models from exploiting spurious correlations. Although model-agnostic meta-learning (MAML) presents as a natural alternative for few-shot transfer learning, the expensive computation due to implicit second-order optimization limits its use on large-scale vision-language models such as CLIP. While much literature has been devoted to exploring alternative optimization strategies, we identify another essential aspect towards effective few-shot transfer learning, task sampling, which is previously only be viewed as part of data pre-processing in MAML. To show the impact of task sampling, we propose a simple algorithm, Model-Agnostic Multitask Fine-tuning (MAMF), which differentiates classical fine-tuning only on uniformly sampling multiple tasks. Despite its simplicity, we show that MAMF consistently outperforms classical fine-tuning on five few-shot vision-language classification tasks. We further show that the effectiveness of the bi-level optimization in MAML is highly sensitive to the zero-shot performance of a task in the context of few-shot vision-language classification. The goal of this paper is to provide new insights on what makes few-shot learning work, and encourage more research into investigating better task sampling strategies.

cs.MM↗

arXiv2022

Provable Domain Generalization via Invariant-Feature Subspace Recovery

Domain generalization asks for models trained over a set of training environments to perform well in unseen test environments. Recently, a series of algorithms such as Invariant Risk Minimization (IRM) has been proposed for domain generalization. However, Rosenfeld et al. (2021) shows that in a simple linear data model, even if non-convexity issues are ignored, IRM and its extensions cannot generalize to unseen environments with less than $d_s+1$ training environments, where $d_s$ is the dimension of the spurious-feature subspace. In this paper, we propose to achieve domain generalization with Invariant-feature Subspace Recovery (ISR). Our first algorithm, ISR-Mean, can identify the subspace spanned by invariant features from the first-order moments of the class-conditional distributions, and achieve provable domain generalization with $d_s+1$ training environments under the data model of Rosenfeld et al. (2021). Our second algorithm, ISR-Cov, further reduces the required number of training environments to $O(1)$ using the information of second-order moments. Notably, unlike IRM, our algorithms bypass non-convexity issues and enjoy global convergence guarantees. Empirically, our ISRs can obtain superior performance compared with IRM on synthetic benchmarks. In addition, on three real-world image and text datasets, we show that both ISRs can be used as simple yet effective post-processing methods to improve the worst-case accuracy of (pre-)trained models against spurious correlations and group shifts.

cs.LG↗

arXiv2022

Understanding Gradual Domain Adaptation: Improved Analysis, Optimal Path and Beyond

The vast majority of existing algorithms for unsupervised domain adaptation (UDA) focus on adapting from a labeled source domain to an unlabeled target domain directly in a one-off way. Gradual domain adaptation (GDA), on the other hand, assumes a path of $(T-1)$ unlabeled intermediate domains bridging the source and target, and aims to provide better generalization in the target domain by leveraging the intermediate ones. Under certain assumptions, Kumar et al. (2020) proposed a simple algorithm, Gradual Self-Training, along with a generalization bound in the order of $e^{O(T)} \left(\varepsilon_0+O\left(\sqrt{log(T)/n}\right)\right)$ for the target domain error, where $\varepsilon_0$ is the source domain error and $n$ is the data size of each domain. Due to the exponential factor, this upper bound becomes vacuous when $T$ is only moderately large. In this work, we analyze gradual self-training under more general and relaxed assumptions, and prove a significantly improved generalization bound as $\varepsilon_0+ O \left(TΔ+ T/\sqrt{n}\right) + \widetilde{O}\left(1/\sqrt{nT}\right)$, where $Δ$ is the average distributional distance between consecutive domains. Compared with the existing bound with an exponential dependency on $T$ as a multiplicative factor, our bound only depends on $T$ linearly and additively. Perhaps more interestingly, our result implies the existence of an optimal choice of $T$ that minimizes the generalization error, and it also naturally suggests an optimal way to construct the path of intermediate domains so as to minimize the accumulative path length $TΔ$ between the source and target. To corroborate the implications of our theory, we examine gradual self-training on multiple semi-synthetic and real datasets, which confirms our findings. We believe our insights provide a path forward toward the design of future GDA algorithms.

cs.LG↗

arXiv2022

Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning

Contrastive self-supervised learning (SSL) learns an embedding space that maps similar data pairs closer and dissimilar data pairs farther apart. Despite its success, one issue has been overlooked: the fairness aspect of representations learned using contrastive SSL. Without mitigation, contrastive SSL techniques can incorporate sensitive information such as gender or race and cause potentially unfair predictions on downstream tasks. In this paper, we propose a Conditional Contrastive Learning (CCL) approach to improve the fairness of contrastive SSL methods. Our approach samples positive and negative pairs from distributions conditioning on the sensitive attribute, or empirically speaking, sampling positive and negative pairs from the same gender or the same race. We show that our approach provably maximizes the conditional mutual information between the learned representations of the positive pairs, and reduces the effect of the sensitive attribute by taking it as the conditional variable. On seven fairness and vision datasets, we empirically demonstrate that the proposed approach achieves state-of-the-art downstream performances compared to unsupervised baselines and significantly improves the fairness of contrastive SSL models on multiple fairness metrics.

cs.LG↗