Searcharxiv⌕ Search

arXiv subjects

Jiawei Zhou

Publications and source records attributed to Jiawei Zhou.

At least 109 records · Page 6Linked to original sources
arXiv2024

Cohomology of minimal Sullivan algebras of non-finite type and their realizations

We prove that the morphisms from a minimal Sullivan algebra $ΛV$ to $A_{PL}(|ΛV|)$, the algebra of polynomial differential forms on its realization, can be quasi-isomorphic if and only if the cohomology $H(ΛV)$ is of finite type. Importantly, $ΛV$ itself need not be of finite type. For example, it can be the minimal Sullivan model of the wedge sum of a circle and a sphere. This provides a negative answer to a question posed by Félix, Halperin, and Thomas. Furthermore, we study the spaces whose homotopy groups are reflected by their minimal Sullivan models as a generalization of Sullivan spaces, and explore which properties of Sullivan spaces can be broadened.

math.AT↗
arXiv2024

A Novel Hybrid Approach for Tornado Prediction in the United States: Kalman-Convolutional BiLSTM with Multi-Head Attention

Tornadoes are among the most intense atmospheric vortex phenomena and pose significant challenges for detection and forecasting. Conventional methods, which heavily depend on ground-based observations and radar data, are limited by issues such as decreased accuracy over greater distances and a high rate of false positives. To address these challenges, this study utilizes the Seamless Hybrid Scan Reflectivity (SHSR) dataset from the Multi-Radar Multi-Sensor (MRMS) system, which integrates data from multiple radar sources to enhance accuracy. A novel hybrid model, the Kalman-Convolutional BiLSTM with Multi-Head Attention, is introduced to improve dynamic state estimation and capture both spatial and temporal dependencies within the data. This model demonstrates superior performance in precision, recall, F1-Score, and accuracy compared to methods such as K-Nearest Neighbors (KNN) and LightGBM. The results highlight the considerable potential of advanced machine learning techniques to improve tornado prediction and reduce false alarm rates. Future research will focus on expanding datasets, exploring innovative model architectures, and incorporating large language models (LLMs) to provide deeper insights. This research introduces a novel model for tornado prediction, offering a robust framework for enhancing forecasting accuracy and public safety.

cs.LG↗
arXiv2024

A Community-Centric Perspective for Characterizing and Detecting Anti-Asian Violence-Provoking Speech

Violence-provoking speech -- speech that implicitly or explicitly promotes violence against the members of the targeted community, contributed to a massive surge in anti-Asian crimes during the pandemic. While previous works have characterized and built tools for detecting other forms of harmful speech, like fear speech and hate speech, our work takes a community-centric approach to studying anti-Asian violence-provoking speech. Using data from ~420k Twitter posts spanning a 3-year duration (January 1, 2020 to February 1, 2023), we develop a codebook to characterize anti-Asian violence-provoking speech and collect a community-crowdsourced dataset to facilitate its large-scale detection using state-of-the-art classifiers. We contrast the capabilities of natural language processing classifiers, ranging from BERT-based to LLM-based classifiers, in detecting violence-provoking speech with their capabilities to detect anti-Asian hateful speech. In contrast to prior work that has demonstrated the effectiveness of such classifiers in detecting hateful speech ($F_1 = 0.89$), our work shows that accurate and reliable detection of violence-provoking speech is a challenging task ($F_1 = 0.69$). We discuss the implications of our findings, particularly the need for proactive interventions to support Asian communities during public health crises. The resources related to the study are available at https://claws-lab.github.io/violence-provoking-speech/.

cs.CL↗
arXiv2024

MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?

While text-to-image models like DALLE-3 and Stable Diffusion are rapidly proliferating, they often encounter challenges such as hallucination, bias, and the production of unsafe, low-quality output. To effectively address these issues, it is crucial to align these models with desired behaviors based on feedback from a multimodal judge. Despite their significance, current multimodal judges frequently undergo inadequate evaluation of their capabilities and limitations, potentially leading to misalignment and unsafe fine-tuning outcomes. To address this issue, we introduce MJ-Bench, a novel benchmark which incorporates a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: alignment, safety, image quality, and bias. Specifically, we evaluate a large variety of multimodal judges including smaller-sized CLIP-based scoring models, open-source VLMs (e.g. LLaVA family), and close-source VLMs (e.g. GPT-4o, Claude 3) on each decomposed subcategory of our preference dataset. Experiments reveal that close-source VLMs generally provide better feedback, with GPT-4o outperforming other judges in average. Compared with open-source VLMs, smaller-sized scoring models can provide better feedback regarding text-image alignment and image quality, while VLMs provide more accurate feedback regarding safety and generation bias due to their stronger reasoning capabilities. Further studies in feedback scale reveal that VLM judges can generally provide more accurate and stable feedback in natural language (Likert-scale) than numerical scales. Notably, human evaluations on end-to-end fine-tuned models using separate feedback from these multimodal judges provide similar conclusions, further confirming the effectiveness of MJ-Bench. All data, code, models are available at https://huggingface.co/MJ-Bench.

cs.CV↗
arXiv2024

HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding

While large vision-language models (LVLMs) have demonstrated impressive capabilities in interpreting multi-modal contexts, they invariably suffer from object hallucinations (OH). We introduce HALC, a novel decoding algorithm designed to mitigate OH in LVLMs. HALC leverages distinct fine-grained optimal visual information in vision-language tasks and operates on both local and global contexts simultaneously. Specifically, HALC integrates a robust auto-focal grounding mechanism (locally) to correct hallucinated tokens on the fly, and a specialized beam search algorithm (globally) to significantly reduce OH while preserving text generation quality. Additionally, HALC can be integrated into any LVLMs as a plug-and-play module without extra training. Extensive experimental studies demonstrate the effectiveness of HALC in reducing OH, outperforming state-of-the-arts across four benchmarks.

cs.CV↗
arXiv2024

InfRS: Incremental Few-Shot Object Detection in Remote Sensing Images

Recently, the field of few-shot detection within remote sensing imagery has witnessed significant advancements. Despite these progresses, the capacity for continuous conceptual learning still poses a significant challenge to existing methodologies. In this paper, we explore the intricate task of incremental few-shot object detection in remote sensing images. We introduce a pioneering fine-tuningbased technique, termed InfRS, designed to facilitate the incremental learning of novel classes using a restricted set of examples, while concurrently preserving the performance on established base classes without the need to revisit previous datasets. Specifically, we pretrain the model using abundant data from base classes and then generate a set of class-wise prototypes that represent the intrinsic characteristics of the data. In the incremental learning stage, we introduce a Hybrid Prototypical Contrastive (HPC) encoding module for learning discriminative representations. Furthermore, we develop a prototypical calibration strategy based on the Wasserstein distance to mitigate the catastrophic forgetting problem. Comprehensive evaluations on the NWPU VHR-10 and DIOR datasets demonstrate that our model can effectively solve the iFSOD problem in remote sensing images. Code will be released.

cs.CV↗
arXiv2024

Formality of Sphere Bundles

We study the formality of orientable sphere bundles over connected compact manifolds. When the base manifold is formal, we prove that the formality of the bundle is equivalent to the vanishing of the Bianchi-Massey tensor introduced by Crowley-Nordström. As an example, this implies that the unit tangent bundle over a formal manifold can only be formal when the base manifold has vanishing Euler characteristic or a rational cohomology ring generated by one element. When the base manifold is not formal, we give an obstruction to the formality of sphere bundles whose Euler class is reducible.

math.AT↗
arXiv2024

LangSplat: 3D Language Gaussian Splatting

Humans live in a 3D world and commonly use natural language to interact with a 3D scene. Modeling a 3D language field to support open-ended language queries in 3D has gained increasing attention recently. This paper introduces LangSplat, which constructs a 3D language field that enables precise and efficient open-vocabulary querying within 3D spaces. Unlike existing methods that ground CLIP language embeddings in a NeRF model, LangSplat advances the field by utilizing a collection of 3D Gaussians, each encoding language features distilled from CLIP, to represent the language field. By employing a tile-based splatting technique for rendering language features, we circumvent the costly rendering process inherent in NeRF. Instead of directly learning CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and then learns language features on the scene-specific latent space, thereby alleviating substantial memory demands imposed by explicit modeling. Existing methods struggle with imprecise and vague 3D language fields, which fail to discern clear boundaries between objects. We delve into this issue and propose to learn hierarchical semantics using SAM, thereby eliminating the need for extensively querying the language field across various scales and the regularization of DINO features. Extensive experimental results show that LangSplat significantly outperforms the previous state-of-the-art method LERF by a large margin. Notably, LangSplat is extremely efficient, achieving a 199 $\times$ speedup compared to LERF at the resolution of 1440 $\times$ 1080. We strongly recommend readers to check out our video results at https://langsplat.github.io/

cs.CV↗
arXiv2024

Few-shot Oriented Object Detection with Memorable Contrastive Learning in Remote Sensing Images

Few-shot object detection (FSOD) has garnered significant research attention in the field of remote sensing due to its ability to reduce the dependency on large amounts of annotated data. However, two challenges persist in this area: (1) axis-aligned proposals, which can result in misalignment for arbitrarily oriented objects, and (2) the scarcity of annotated data still limits the performance for unseen object categories. To address these issues, we propose a novel FSOD method for remote sensing images called Few-shot Oriented object detection with Memorable Contrastive learning (FOMC). Specifically, we employ oriented bounding boxes instead of traditional horizontal bounding boxes to learn a better feature representation for arbitrary-oriented aerial objects, leading to enhanced detection performance. To the best of our knowledge, we are the first to address oriented object detection in the few-shot setting for remote sensing images. To address the challenging issue of object misclassification, we introduce a supervised contrastive learning module with a dynamically updated memory bank. This module enables the use of large batches of negative samples and enhances the model's capability to learn discriminative features for unseen classes. We conduct comprehensive experiments on the DOTA and HRSC2016 datasets, and our model achieves state-of-the-art performance on the few-shot oriented object detection task. Code and pretrained models will be released.

cs.CV↗
arXiv2024

Retrieval-based Disentangled Representation Learning with Natural Language Supervision

Disentangled representation learning remains challenging as the underlying factors of variation in the data do not naturally exist. The inherent complexity of real-world data makes it unfeasible to exhaustively enumerate and encapsulate all its variations within a finite set of factors. However, it is worth noting that most real-world data have linguistic equivalents, typically in the form of textual descriptions. These linguistic counterparts can represent the data and effortlessly decomposed into distinct tokens. In light of this, we present Vocabulary Disentangled Retrieval (VDR), a retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning. Our approach employ a bi-encoder model to represent both data and natural language in a vocabulary space, enabling the model to distinguish dimensions that capture intrinsic characteristics within data through its natural language counterpart, thus facilitating disentanglement. We extensively assess the performance of VDR across 15 retrieval benchmark datasets, covering text-to-text and cross-modal retrieval scenarios, as well as human evaluation. Our experimental results compellingly demonstrate the superiority of VDR over previous bi-encoder retrievers with comparable model size and training costs, achieving an impressive 8.7% improvement in NDCG@10 on the BEIR benchmark, a 5.3% increase on MS COCO, and a 6.0% increase on Flickr30k in terms of mean recall in the zero-shot setting. Moreover, The results from human evaluation indicate that interpretability of our method is on par with SOTA captioning models.

cs.CL↗
arXiv2023

Quick Back-Translation for Unsupervised Machine Translation

The field of unsupervised machine translation has seen significant advancement from the marriage of the Transformer and the back-translation algorithm. The Transformer is a powerful generative model, and back-translation leverages Transformer's high-quality translations for iterative self-improvement. However, the Transformer is encumbered by the run-time of autoregressive inference during back-translation, and back-translation is limited by a lack of synthetic data efficiency. We propose a two-for-one improvement to Transformer back-translation: Quick Back-Translation (QBT). QBT re-purposes the encoder as a generative model, and uses encoder-generated sequences to train the decoder in conjunction with the original autoregressive back-translation step, improving data throughput and utilization. Experiments on various WMT benchmarks demonstrate that a relatively small number of refining steps of QBT improve current unsupervised machine translation models, and that QBT dramatically outperforms standard back-translation only method in terms of training efficiency for comparable translation qualities.

cs.CL↗
arXiv2023

Defects Vibrations Engineering for Enhancing Interfacial Thermal Transport

To push upper boundaries of effective thermal conductivity in polymer composites, a fundamental understanding of thermal transport mechanisms is crucial. Although there is intensive simulation research, systematic experimental investigation on thermal transport in polymer composites is limited. To better understand thermal transport processes, we design polymer composites with perfect fillers (graphite) and defective fillers (graphite oxide); we choose polar polyvinyl alcohol (PVA) as a matrix model; and we identify how thermal transport occurs across heterogeneous interfaces. Measured thermal conductivities of in PVA/defective filler composites is higher than those of PVA/perfect filler composites, while measured thermal conductivities in defective fillers are lower than those of perfect fillers. An effective quantum mechanical model is developed, showing that the vibrational state of the defective level plays a critical role in enhancing the thermal conductivity with increased defect concentration. Our experimental and model results have suggested that defects in polymer composites may enhance thermal transport in polymer composites by promoting vibrational resonant couplings.

physics.app-ph↗
arXiv2023

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation

There has been a growing interest in developing multimodal machine translation (MMT) systems that enhance neural machine translation (NMT) with visual knowledge. This problem setup involves using images as auxiliary information during training, and more recently, eliminating their use during inference. Towards this end, previous works face a challenge in training powerful MMT models from scratch due to the scarcity of annotated multilingual vision-language data, especially for low-resource languages. Simultaneously, there has been an influx of multilingual pre-trained models for NMT and multimodal pre-trained models for vision-language tasks, primarily in English, which have shown exceptional generalisation ability. However, these are not directly applicable to MMT since they do not provide aligned multimodal multilingual features for generative tasks. To alleviate this issue, instead of designing complex modules for MMT, we propose CLIPTrans, which simply adapts the independently pre-trained multimodal M-CLIP and the multilingual mBART. In order to align their embedding spaces, mBART is conditioned on the M-CLIP features by a prefix sequence generated through a lightweight mapping network. We train this in a two-stage pipeline which warms up the model with image captioning before the actual translation task. Through experiments, we demonstrate the merits of this framework and consequently push forward the state-of-the-art across standard benchmarks by an average of +2.67 BLEU. The code can be found at www.github.com/devaansh100/CLIPTrans.

cs.CV↗
arXiv2023

3D Face Arbitrary Style Transfer

Style transfer of 3D faces has gained more and more attention. However, previous methods mainly use images of artistic faces for style transfer while ignoring arbitrary style images such as abstract paintings. To solve this problem, we propose a novel method, namely Face-guided Dual Style Transfer (FDST). To begin with, FDST employs a 3D decoupling module to separate facial geometry and texture. Then we propose a style fusion strategy for facial geometry. Subsequently, we design an optimization-based DDSG mechanism for textures that can guide the style transfer by two style images. Besides the normal style image input, DDSG can utilize the original face input as another style input as the face prior. By this means, high-quality face arbitrary style transfer results can be obtained. Furthermore, FDST can be applied in many downstream tasks, including region-controllable style transfer, high-fidelity face texture reconstruction, large-pose face reconstruction, and artistic face reconstruction. Comprehensive quantitative and qualitative results show that our method can achieve comparable performance. All source codes and pre-trained weights will be released to the public.

cs.CV↗
arXiv2022

Symplectically Flat Connections and Their Functionals

We continue our study of symplectically flat bundles. We broaden the notion of symplectically flat connections on symplectic manifolds to $ζ$-flat connections on smooth manifolds. These connections on principal bundles can be represented by maps from an extension of the base's fundamental group to the structure group. We introduce functionals with zeroes being symplectically flat connections and study their critical points. Such functionals lead to novel geometric flows. We also describe some characteristic classes of the $ζ$-flat bundles.

math.SG↗
arXiv2022

$A_\infty$-Minimal Model on Differential Graded Algebras

The rational homotopy type of a differential graded algebra (DGA) can be represented by a family of tensors on its cohomology, which constitute an $A_\infty$-minimal model of this DGA. When only the cohomology is needed to determine the rational homotopy type, then the DGA is called formal. By a theorem of Miller, a compact $k$-connected manifold is formal if its dimension is not greater than $4k+2$. We expand this theorem and a result of Crowley-Nordström to prove that if the dimension of a compact $k$-connected manifold $N\leq (l+1)k+2$, then its de Rham complex has an $A_\infty$-minimal model with $m_p=0$ for all $p\geq l$. Separately, for an odd-dimensional sphere bundle over a formal manifold, we prove that its de Rham complex has an $A_\infty$-minimal model with only $m_2$ and $m_3$ non-trivial. In the special case of a circle bundle over a formal symplectic manifold satisfying the hard Lefschetz property, we give a necessary condition for formality which becomes sufficient when the base symplectic manifold is of dimension six or less.

math.AT↗
arXiv2022

Symplectic Flatness and Twisted Primitive Cohomology

We introduce the notion of symplectic flatness for connections and fiber bundles over symplectic manifolds. Given an $A_\infty$-algebra, we present a flatness condition that enables the twisting of the differential complex associated with the $A_\infty$-algebra. The symplectic flatness condition arises from twisting the $A_\infty$-algebra of differential forms constructed by Tsai, Tseng and Yau. When the symplectic manifold is equipped with a compatible metric, the symplectic flat connections represent a special subclass of Yang-Mills connections. We further study the cohomologies of the twisted differential complex and give a simple vanishing theorem for them.

math.SG↗