arXiv subjects

Tao Gong

Publications and source records attributed to Tao Gong.

At least 55 records · Page 3Linked to original sources

arXiv2024

Towards More Unified In-context Visual Understanding

The rapid advancement of large language models (LLMs) has accelerated the emergence of in-context learning (ICL) as a cutting-edge approach in the natural language processing domain. Recently, ICL has been employed in visual understanding tasks, such as semantic segmentation and image captioning, yielding promising results. However, existing visual ICL framework can not enable producing content across multiple modalities, which limits their potential usage scenarios. To address this issue, we present a new ICL framework for visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space, structured as interleaved in-context sequences. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them, facilitating in-context learning. Thanks to this design, the model is capable of handling in-context vision understanding tasks with multimodal output in a unified pipeline.Experimental results demonstrate that our model achieves competitive performance compared with specialized models and previous ICL baselines. Overall, our research takes a further step toward unified multimodal in-context learning.

cs.CV↗

arXiv2024

Casimir repulsion with biased semiconductors

Quantum and thermal fluctuations are fundamental to a plethora of phenomena within quantum optics, including the Casimir effect that acts between closely separated surfaces typically found in MEMS and NEMS devices. Particularly promising for engineering and harnessing these forces are systems out of thermal equilibrium. Recently, semiconductors with external bias have been proposed to study the nonequilibrium Casimir force. Here, we explore systems involving moderately biased semiconductors that exhibit strong repulsive Casimir forces, and we determine the effects of bias voltage, semiconductor bandgap energy, and separation for experimentally accessible configurations. Modes emitted from the semiconductors exert a repulsive force on a near surface that overcomes the attractive equilibrium Casimir force contribution at submicron distances. For the geometry of two parallel planes, those modes undergo Fabry-Pérot interference resulting in an oscillatory force behavior as a function of separation. Utilizing the proximity-force approximation, we predict that the repulsive force exerted on a gold sphere is well within the accuracy of typical Casimir force experiments. Our work opens up new possibilities of controlling forces at the nano- and micrometer scale with applications in sensing and actuation in nanotechnology.

quant-ph↗

arXiv2024

Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues

How to effectively interact audio with vision has garnered considerable interest within the multi-modality research field. Recently, a novel audio-visual segmentation (AVS) task has been proposed, aiming to segment the sounding objects in video frames under the guidance of audio cues. However, most existing AVS methods are hindered by a modality imbalance where the visual features tend to dominate those of the audio modality, due to a unidirectional and insufficient integration of audio cues. This imbalance skews the feature representation towards the visual aspect, impeding the learning of joint audio-visual representations and potentially causing segmentation inaccuracies. To address this issue, we propose AVSAC. Our approach features a Bidirectional Audio-Visual Decoder (BAVD) with integrated bidirectional bridges, enhancing audio cues and fostering continuous interplay between audio and visual modalities. This bidirectional interaction narrows the modality imbalance, facilitating more effective learning of integrated audio-visual representations. Additionally, we present a strategy for audio-visual frame-wise synchrony as fine-grained guidance of BAVD. This strategy enhances the share of auditory components in visual features, contributing to a more balanced audio-visual representation learning. Extensive experiments show that our method attains new benchmarks in AVS performance.

cs.CV↗

arXiv2023

CloudBrain-MRS: An Intelligent Cloud Computing Platform for in vivo Magnetic Resonance Spectroscopy Preprocessing, Quantification, and Analysis

Magnetic resonance spectroscopy (MRS) is an important clinical imaging method for diagnosis of diseases. MRS spectrum is used to observe the signal intensity of metabolites or further infer their concentrations. Although the magnetic resonance vendors commonly provide basic functions of spectra plots and metabolite quantification, the widespread clinical research of MRS is still limited due to the lack of easy-to-use processing software or platform. To address this issue, we have developed CloudBrain-MRS, a cloud-based online platform that provides powerful hardware and advanced algorithms. The platform can be accessed simply through a web browser, without the need of any program installation on the user side. CloudBrain-MRS also integrates the classic LCModel and advanced artificial intelligence algorithms and supports batch preprocessing, quantification, and analysis of MRS data from different vendors. Additionally, the platform offers useful functions: 1) Automatically statistical analysis to find biomarkers for diseases; 2) Consistency verification between the classic and artificial intelligence quantification algorithms; 3) Colorful three-dimensional visualization for easy observation of individual metabolite spectrum. Last, both healthy and mild cognitive impairment patient data are used to demonstrate the functions of the platform. To the best of our knowledge, this is the first cloud computing platform for in vivo MRS with artificial intelligence processing. We have shared our cloud platform at MRSHub, providing free access and service for two years. Please visit https://mrshub.org/software_all/#CloudBrain-MRS or https://csrc.xmu.edu.cn/CloudBrain.html.

eess.SP↗

arXiv2023

MultiModal-GPT: A Vision and Language Model for Dialogue with Humans

We present a vision and language model named MultiModal-GPT to conduct multi-round dialogue with humans. MultiModal-GPT can follow various instructions from humans, such as generating a detailed caption, counting the number of interested objects, and answering general questions from users. MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with Low-rank Adapter (LoRA) added both in the cross-attention part and the self-attention part of the language model. We first construct instruction templates with vision and language data for multi-modality instruction tuning to make the model understand and follow human instructions. We find the quality of training data is vital for the dialogue performance, where few data containing short answers can lead the model to respond shortly to any instructions. To further enhance the ability to chat with humans of the MultiModal-GPT, we utilize language-only instruction-following data to train the MultiModal-GPT jointly. The joint training of language-only and visual-language instructions with the \emph{same} instruction template effectively improves dialogue performance. Various demos show the ability of continuous dialogue of MultiModal-GPT with humans. Code, dataset, and demo are at https://github.com/open-mmlab/Multimodal-GPT

cs.CV↗

arXiv2023

StrongSORT: Make DeepSORT Great Again

Recently, Multi-Object Tracking (MOT) has attracted rising attention, and accordingly, remarkable progresses have been achieved. However, the existing methods tend to use various basic models (e.g, detector and embedding model), and different training or inference tricks, etc. As a result, the construction of a good baseline for a fair comparison is essential. In this paper, a classic tracker, i.e., DeepSORT, is first revisited, and then is significantly improved from multiple perspectives such as object detection, feature embedding, and trajectory association. The proposed tracker, named StrongSORT, contributes a strong and fair baseline for the MOT community. Moreover, two lightweight and plug-and-play algorithms are proposed to address two inherent "missing" problems of MOT: missing association and missing detection. Specifically, unlike most methods, which associate short tracklets into complete trajectories at high computation complexity, we propose an appearance-free link model (AFLink) to perform global association without appearance information, and achieve a good balance between speed and accuracy. Furthermore, we propose a Gaussian-smoothed interpolation (GSI) based on Gaussian process regression to relieve the missing detection. AFLink and GSI can be easily plugged into various trackers with a negligible extra computational cost (1.7 ms and 7.1 ms per image, respectively, on MOT17). Finally, by fusing StrongSORT with AFLink and GSI, the final tracker (StrongSORT++) achieves state-of-the-art results on multiple public benchmarks, i.e., MOT17, MOT20, DanceTrack and KITTI. Codes are available at https://github.com/dyhBUPT/StrongSORT and https://github.com/open-mmlab/mmtracking.

cs.CV↗

arXiv2022

Electrically switchable Casimir forces using transparent conductive oxides

Casimir forces between charge-neutral bodies originate from quantum vacuum fluctuations of electromagnetic fields, which exhibit a critical dependence on material's electromagnetic properties. Over the years, in-situ modulation of material's optical properties has been enabled through various means and has been widely exploited in a plethora of applications such as electro-optical modulation, transient color generation, bio- or chemical sensing, etc. Yet Casimir force modulation has been hindered by difficulty in achieving high modulation signals due to the broadband nature of the Casimir interaction. Here we propose and investigate two configurations that allow for in-situ modulation of Casimir forces through electrical gating of a metal-insulator-semiconductor (MIS) junction comprised of transparent conductive oxide (TCO) materials. By switching the gate voltage on and off, a force modulation of > 400 pN is predicted due to substantive charge carrier accumulation in the TCO layer, which can be easily measured using state-of-the-art force measurement techniques in an atomic force microscope (AFM). We further examine the influence of the oxide layer thickness on the force modulation, suggesting the importance of the fine control of the oxide layer deposition. Our work provides a promising pathway for modulating the Casimir effect in-situ with experimentally measurable force contrast.

physics.app-ph↗

arXiv2022

The Effect of Epsilon-Near-Zero (ENZ) Modes on the Casimir Interaction between Ultrathin Films

Vacuum fluctuation-induced interactions between macroscopic metallic objects result in an attractive force between them, a phenomenon known as the Casimir effect. This force is the result of both plasmonic and photonic modes. For very thin films, field penetration through the films will modify the allowed modes. Here, we investigate the Casimir interaction between two ultrathin films from the perspective of the force distribution over real frequencies for the first time and find pronounced repulsive contributions to the force due to the highly confined and nearly dispersion-free epsilon-near-zero (ENZ) modes that only exist in ultrathin films. These contributions are found to persistently occur around the ENZ frequency of the film and are irrespective of the inter-film separation. We further associate the ENZ modes with a striking thickness dependence in the averaged force density for conductive thin films, a metric signifying a thin-film's acceleration due to Casimir effect. Our results shed light on the role of the unique vacuum fluctuation modes existing in ultrathin ENZ materials, which may offer significant potential for engineering the motion of objects in nanomechanical systems.

quant-ph↗

arXiv2022

Observation of intrinsic crystal phase in bare CrI3 ferromagnetism

Intrinsic structural phase is a crucial foundation for the fundamental physical properties, and for creating innovative devices with unprecedented performances and unique functionalities. Long-range ferromagnetic orders of van der Waals CrI3 are strongly tied with interlayer stacking orders. However, the intrinsic structure of few-layer CrI3 still remains elusive; the predicted monoclinic phase has not yet been experimentally detected in bare few-layer CrI3. Here we uncover the intrinsic structure of few-layer CrI3 with interlayer antiferromagnetic coupling, which unambiguously show monoclinic stacking in both bare and hBN-encapsulated bilayer and tri-five-layer CrI3 throughout an entire temperature range from 300 to 10 K. An exotic spring damping effect from hBN encapsulation layers is experimentally observed in hBN/CrI3/hBN heterostructures, which partly hinders interlayer sliding of CrI3. This work demonstrates the intrinsic monoclinic crystal phase of few-layer CrI3 and associated correlation with magnetic orders, opening up numerous opportunities for creating magnetic texture by stacking design.

cond-mat.mtrl-sci↗

arXiv2022

Radiative energy bandgap of nanostructures coupled with quantum emitters around the epsilon-near-zero (ENZ) frequency

Epsilon-near-zero (ENZ) materials have been demonstrated to exhibit unique electromagnetic properties. Here we propose the concept of radiative energy bandgap for an ENZ nanoparticle coupled with a quantum emitter (QE). The radiative emission of the coupled QE-nanoparticle can be significantly suppressed around the ENZ frequency and substantially enhanced otherwise, yielding an effective energy bandgap for radiation. This suppression is effectively invariant with respect to the particle size and is therefore an intrinsic property of the ENZ material. Our concept also heralds an alternative pathway to quench the emission from a QE, which may find potential application in quantum information storage.

physics.optics↗

arXiv2021

Engineering Casimir interactions with epsilon-near-zero materials

In this paper we theoretically demonstrate the tunability of the Casimir force both in sign and magnitude between parallel plates coated with dispersive materials. We show that this force, existing between uncharged plates, can be tuned by carefully choosing the value of the plasma frequency (i.e., the epsilon-near-zero frequency) of the coating in the neighborhood of the resonance frequency of the cavity. The coating layer enables a continuous variation of the force between four limiting values when a coating is placed on each plate. We explore the consequences of such variation when pairs of electric and magnetic conductors (i.e. low and high impedance surfaces) are used as substrates on either side, showing that this continuous variation results in changes in the sign of the force, leading to both stable and unstable conditions, which could find interesting potential applications in nanomechanics including nanoparticle tweezing.

quant-ph↗

arXiv2021

Temporal RoI Align for Video Object Recognition

Video object detection is challenging in the presence of appearance deterioration in certain video frames. Therefore, it is a natural choice to aggregate temporal information from other frames of the same video into the current frame. However, RoI Align, as one of the most core procedures of video detectors, still remains extracting features from a single-frame feature map for proposals, making the extracted RoI features lack temporal information from videos. In this work, considering the features of the same object instance are highly similar among frames in a video, a novel Temporal RoI Align operator is proposed to extract features from other frames feature maps for current frame proposals by utilizing feature similarity. The proposed Temporal RoI Align operator can extract temporal information from the entire video for proposals. We integrate it into single-frame video detectors and other state-of-the-art video detectors, and conduct quantitative experiments to demonstrate that the proposed Temporal RoI Align operator can consistently and significantly boost the performance. Besides, the proposed Temporal RoI Align can also be applied into video instance segmentation. Codes are available at https://github.com/open-mmlab/mmtracking

cs.CV↗

arXiv2021

Mining Contextual Information Beyond Image for Semantic Segmentation

This paper studies the context aggregation problem in semantic image segmentation. The existing researches focus on improving the pixel representations by aggregating the contextual information within individual images. Though impressive, these methods neglect the significance of the representations of the pixels of the corresponding class beyond the input image. To address this, this paper proposes to mine the contextual information beyond individual images to further augment the pixel representations. We first set up a feature memory module, which is updated dynamically during training, to store the dataset-level representations of various categories. Then, we learn class probability distribution of each pixel representation under the supervision of the ground-truth segmentation. At last, the representation of each pixel is augmented by aggregating the dataset-level representations based on the corresponding class probability distribution. Furthermore, by utilizing the stored dataset-level representations, we also propose a representation consistent learning strategy to make the classification head better address intra-class compactness and inter-class dispersion. The proposed method could be effortlessly incorporated into existing segmentation frameworks (e.g., FCN, PSPNet, OCRNet and DeepLabV3) and brings consistent performance improvements. Mining contextual information beyond image allows us to report state-of-the-art performance on various benchmarks: ADE20K, LIP, Cityscapes and COCO-Stuff.

cs.CV↗

arXiv2021

Seesaw Loss for Long-Tailed Instance Segmentation

Instance segmentation has witnessed a remarkable progress on class-balanced benchmarks. However, they fail to perform as accurately in real-world scenarios, where the category distribution of objects naturally comes with a long tail. Instances of head classes dominate a long-tailed dataset and they serve as negative samples of tail categories. The overwhelming gradients of negative samples on tail classes lead to a biased learning process for classifiers. Consequently, objects of tail categories are more likely to be misclassified as backgrounds or head categories. To tackle this problem, we propose Seesaw Loss to dynamically re-balance gradients of positive and negative samples for each category, with two complementary factors, i.e., mitigation factor and compensation factor. The mitigation factor reduces punishments to tail categories w.r.t. the ratio of cumulative training instances between different categories. Meanwhile, the compensation factor increases the penalty of misclassified instances to avoid false positives of tail categories. We conduct extensive experiments on Seesaw Loss with mainstream frameworks and different data sampling strategies. With a simple end-to-end training pipeline, Seesaw Loss obtains significant gains over Cross-Entropy Loss, and achieves state-of-the-art performance on LVIS dataset without bells and whistles. Code is available at https://github.com/open-mmlab/mmdetection.

cs.CV↗

arXiv2021

Towards Generalizable and Robust Face Manipulation Detection via Bag-of-local-feature

Over the past several years, in order to solve the problem of malicious abuse of facial manipulation technology, face manipulation detection technology has obtained considerable attention and achieved remarkable progress. However, most existing methods have very impoverished generalization ability and robustness. In this paper, we propose a novel method for face manipulation detection, which can improve the generalization ability and robustness by bag-of-local-feature. Specifically, we extend Transformers using bag-of-feature approach to encode inter-patch relationships, allowing it to learn local forgery features without any explicit supervision. Extensive experiments demonstrate that our method can outperform competing state-of-the-art methods on FaceForensics++, Celeb-DF and DeeperForensics-1.0 datasets.

cs.CV↗

arXiv2020

Quantitative predictions of photo-emission dynamics in metal halide perovskites via machine learning

Metal halide perovskite (MHP) optoelectronics may become a viable alternative to standard Si-based technologies, but the current lack of long-term stability precludes their commercial adoption. Exposure to standard operational stressors (light, temperature, bias, oxygen, and water) often instigate optical and electronic dynamics, calling for a systematic investigation into MHP photophysical processes and the development of quantitative models for their prediction. We resolve the moisture-driven light emission dynamics for both methylammonium lead tribromide and triiodide thin films as a function of relative humidity (rH). With the humidity and photoluminescence time series, we train recurrent neural networks and establish their ability to quantitatively predict the path of future light emission with <11% error over 12 hours. Together, our in situ rH-PL measurements and machine learning forecasting models provide a framework for the rational design of future stable perovskite devices and, thus, a faster transition towards commercial applications.

cond-mat.mtrl-sci↗

arXiv2020

Side-Aware Boundary Localization for More Precise Object Detection

Current object detection frameworks mainly rely on bounding box regression to localize objects. Despite the remarkable progress in recent years, the precision of bounding box regression remains unsatisfactory, hence limiting performance in object detection. We observe that precise localization requires careful placement of each side of the bounding box. However, the mainstream approach, which focuses on predicting centers and sizes, is not the most effective way to accomplish this task, especially when there exists displacements with large variance between the anchors and the targets. In this paper, we propose an alternative approach, named as Side-Aware Boundary Localization (SABL), where each side of the bounding box is respectively localized with a dedicated network branch. To tackle the difficulty of precise localization in the presence of displacements with large variance, we further propose a two-step localization scheme, which first predicts a range of movement through bucket prediction and then pinpoints the precise position within the predicted bucket. We test the proposed method on both two-stage and single-stage detection frameworks. Replacing the standard bounding box regression branch with the proposed design leads to significant improvements on Faster R-CNN, RetinaNet, and Cascade R-CNN, by 3.0%, 1.7%, and 0.9%, respectively. Code is available at https://github.com/open-mmlab/mmdetection.

cs.CV↗

arXiv2020

A Large Scale Urban Surveillance Video Dataset for Multiple-Object Tracking and Behavior Analysis

Multiple-object tracking and behavior analysis have been the essential parts of surveillance video analysis for public security and urban management. With billions of surveillance video captured all over the world, multiple-object tracking and behavior analysis by manual labor are cumbersome and cost expensive. Due to the rapid development of deep learning algorithms in recent years, automatic object tracking and behavior analysis put forward an urgent demand on a large scale well-annotated surveillance video dataset that can reflect the diverse, congested, and complicated scenarios in real applications. This paper introduces an urban surveillance video dataset (USVD) which is by far the largest and most comprehensive. The dataset consists of 16 scenes captured in 7 typical outdoor scenarios: street, crossroads, hospital entrance, school gate, park, pedestrian mall, and public square. Over 200k video frames are annotated carefully, resulting in more than 3:7 million object bounding boxes and about 7:1 thousand trajectories. We further use this dataset to evaluate the performance of typical algorithms for multiple-object tracking and anomaly behavior analysis and explore the robustness of these methods in urban congested scenarios.

cs.CV↗