arXiv subjects

Yi Jiang

Publications and source records attributed to Yi Jiang.

At least 163 records · Page 9Linked to original sources

arXiv2023

InstMove: Instance Motion for Object-centric Video Segmentation

Despite significant efforts, cutting-edge video segmentation methods still remain sensitive to occlusion and rapid movement, due to their reliance on the appearance of objects in the form of object embeddings, which are vulnerable to these disturbances. A common solution is to use optical flow to provide motion information, but essentially it only considers pixel-level motion, which still relies on appearance similarity and hence is often inaccurate under occlusion and fast movement. In this work, we study the instance-level motion and present InstMove, which stands for Instance Motion for Object-centric Video Segmentation. In comparison to pixel-wise motion, InstMove mainly relies on instance-level motion information that is free from image feature embeddings, and features physical interpretations, making it more accurate and robust toward occlusion and fast-moving objects. To better fit in with the video segmentation tasks, InstMove uses instance masks to model the physical presence of an object and learns the dynamic model through a memory network to predict its position and shape in the next frame. With only a few lines of code, InstMove can be integrated into current SOTA methods for three different video segmentation tasks and boost their performance. Specifically, we improve the previous arts by 1.5 AP on OVIS dataset, which features heavy occlusions, and 4.9 AP on YouTubeVIS-Long dataset, which mainly contains fast-moving objects. These results suggest that instance-level motion is robust and accurate, and hence serving as a powerful solution in complex scenarios for object-centric video segmentation.

cs.CV↗

arXiv2023

Correlating Chemical Reaction and Mass Transport in Hydrogen-based Direct Reduction of Iron Oxide

Steelmaking contributes 8% to the total CO2 emissions globally, primarily due to coal-based iron ore reduction. Clean hydrogen-based ironmaking has variable performance because the dominant gas-solid reduction mechanism is set by the defects and pores inside the mm-nm sized oxide particles that change significantly as the reaction progresses. While these governing dynamics are essential to establish continuous flow of iron and its ores through reactors, the direct link between agglomeration and chemistry is still contested due to missing measurements. In this work, we directly measure the connection between chemistry and agglomeration in the smallest iron oxides relevant to magnetite ores. Using synthesized spherical 10-nm magnetite particles reacting in H2, we resolve the formation and consumption of wüstite (FeO) - the step most commonly attributed to agglomeration. Using X-ray scattering and microscopy, we resolve crystallographic anisotropy in the rate of the initial reaction, which becomes isotropic as the material sinters. Complementing with imaging, we demonstrate how the particles self-assemble, subsequently react and sinter into ~100x oblong grains. Our insights into how morphologically uniform iron oxide particles react and agglomerate H2 reduction enable future size-dependent models to effectively describe the multiscale iron ore reduction.

cond-mat.mtrl-sci↗

arXiv2023

Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling

We identify and overcome two key obstacles in extending the success of BERT-style pre-training, or the masked image modeling, to convolutional networks (convnets): (i) convolution operation cannot handle irregular, random-masked input images; (ii) the single-scale nature of BERT pre-training is inconsistent with convnet's hierarchical structure. For (i), we treat unmasked pixels as sparse voxels of 3D point clouds and use sparse convolution to encode. This is the first use of sparse convolution for 2D masked modeling. For (ii), we develop a hierarchical decoder to reconstruct images from multi-scale encoded features. Our method called Sparse masKed modeling (SparK) is general: it can be used directly on any convolutional model without backbone modifications. We validate it on both classical (ResNet) and modern (ConvNeXt) models: on three downstream tasks, it surpasses both state-of-the-art contrastive learning and transformer-based masked modeling by similarly large margins (around +1.0%). Improvements on object detection and instance segmentation are more substantial (up to +3.5%), verifying the strong transferability of features learned. We also find its favorable scaling behavior by observing more gains on larger models. All this evidence reveals a promising future of generative pre-training on convnets. Codes and models are released at https://github.com/keyu-tian/SparK.

cs.CV↗

arXiv2022

Prototype matching: children's preference for forming scientific concepts

Inspired by a sample lesson, this paper studies and discusses children's preferences in learning scientific concepts. In a "Dissolution" lesson, one of the students took the demonstration experiment of "carmine dissolves in Water" demonstrated by the teacher as the prototype to judge whether a new phenomenon belongs to dissolution, instead of analyzing and judging the phenomenon by using the dissolution definition. Therefore, we propose a conjecture that "prototype matching" may be a more preferred way for children to learn concepts than thinking through inquiry experiment, analysis, deduction, etc. To this end, we conducted a targeted test on 160 fifth grade students (all of whom had learned this lesson) from a primary school in Beijing, and used goodness of fit test to statistically analyze the results. The results showed that: 1. the Chi square of the general result is 73.865, P<0.001, indicating that children did have obvious prototype preference; 2. We "tampered" some of the prototypes, that is, they looked like the prototypes that the teacher had told students, but they were actually wrong. However, the results showed that children still preferred these so-called "prototypes" (chi square is 21.823, P<0.001). Conclusion: 1. Children have an obvious preference for "prototype matching" in scientific concept learning, which is not only obviously deviated from the current general understanding of science education that emphasizes discovery/inquiry construction, but also points out that there may be a priority relationship among various ways of concept organization (such as definition theory, prototype theory, schema theory, etc.). 2. Children's preference for prototypes seems to be unthinking, and they will not identify the authenticity of prototypes, which is particularly noteworthy in front-line teaching.

q-bio.NC↗

arXiv2022

Multi-view deep learning based molecule design and structural optimization accelerates the SARS-CoV-2 inhibitor discovery

In this work, we propose MEDICO, a Multi-viEw Deep generative model for molecule generation, structural optimization, and the SARS-CoV-2 Inhibitor disCOvery. To the best of our knowledge, MEDICO is the first-of-this-kind graph generative model that can generate molecular graphs similar to the structure of targeted molecules, with a multi-view representation learning framework to sufficiently and adaptively learn comprehensive structural semantics from targeted molecular topology and geometry. We show that our MEDICO significantly outperforms the state-of-the-art methods in generating valid, unique, and novel molecules under benchmarking comparisons. In particular, we showcase the multi-view deep learning model enables us to generate not only the molecules structurally similar to the targeted molecules but also the molecules with desired chemical properties, demonstrating the strong capability of our model in exploring the chemical space deeply. Moreover, case study results on targeted molecule generation for the SARS-CoV-2 main protease (Mpro) show that by integrating molecule docking into our model as chemical priori, we successfully generate new small molecules with desired drug-like properties for the Mpro, potentially accelerating the de novo design of Covid-19 drugs. Further, we apply MEDICO to the structural optimization of three well-known Mpro inhibitors (N3, 11a, and GC376) and achieve ~88% improvement in their binding affinity to Mpro, demonstrating the application value of our model for the development of therapeutics for SARS-CoV-2 infection.

cs.LG↗

arXiv2022

Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time

Noninvasive X-ray imaging of nanoscale three-dimensional objects, e.g. integrated circuits (ICs), generally requires two types of scanning: ptychographic, which is translational and returns estimates of complex electromagnetic field through ICs; and tomographic scanning, which collects complex field projections from multiple angles. Here, we present Attentional Ptycho-Tomography (APT), an approach trained to provide accurate reconstructions of ICs despite incomplete measurements, using a dramatically reduced amount of angular scanning. Training process includes regularizing priors based on typical IC patterns and the physics of X-ray propagation. We demonstrate that APT with 12-time reduced angles achieves fidelity comparable to the gold standard with the original set of angles. With the same set of reduced angles, APT also outperforms baseline reconstruction methods. In our experiments, APT achieves 108-time aggregate reduction in data acquisition and computation without compromising quality. We expect our physics-assisted machine learning framework could also be applied to other branches of nanoscale imaging.

eess.IV↗

arXiv2022

Learning Object-Language Alignments for Open-Vocabulary Object Detection

Existing object detection methods are bounded in a fixed-set vocabulary by costly labeled data. When dealing with novel categories, the model has to be retrained with more bounding box annotations. Natural language supervision is an attractive alternative for its annotation-free attributes and broader object concepts. However, learning open-vocabulary object detection from language is challenging since image-text pairs do not contain fine-grained object-language alignments. Previous solutions rely on either expensive grounding annotations or distilling classification-oriented vision models. In this paper, we propose a novel open-vocabulary object detection framework directly learning from image-text pair data. We formulate object-language alignment as a set matching problem between a set of image region features and a set of word embeddings. It enables us to train an open-vocabulary object detector on image-text pairs in a much simple and effective way. Extensive experiments on two benchmark datasets, COCO and LVIS, demonstrate our superior performance over the competing approaches on novel categories, e.g. achieving 32.0% mAP on COCO and 21.7% mask mAP on LVIS. Code is available at: https://github.com/clin1223/VLDet.

cs.CV↗

arXiv2022

RIS-Assisted Self-Interference Mitigation for In-Band Full-Duplex Transceivers

The wireless in-band full-duplex (IBFD) technology can in theory double the system capacity over the conventional frequency division duplex (FDD) or time-division duplex (TDD) alternatives. But the strong self-interference of the IBFD can cause excessive quantization noise in the analog-to-digital converters (ADC), which represents the hurdle for its real implementation. In this paper, we consider employing a reconfigurable intelligent surface (RIS) for IBFD communications. While the BS transmits and receives the signals to and from the users simultaneously on the same frequency band, it can adjust the reflection coefficients of the RIS to configure the wireless channel so that the self-interference of the BS is sufficiently mitigated in the propagation domain. Taking the impact of the quantization noise into account, we propose to jointly design the downlink (DL) precoding matrix and the RIS coefficients to maximize the sum of uplink (UL) and DL rates. The effectiveness of the proposed RIS-assisted in-band full-duplex (RAIBFD) system is verified by simulation studies, even taking into considerations that the phases of the RIS have only finite resolution.

eess.SP↗

arXiv2022

The Runner-up Solution for YouTube-VIS Long Video Challenge 2022

This technical report describes our 2nd-place solution for the ECCV 2022 YouTube-VIS Long Video Challenge. We adopt the previously proposed online video instance segmentation method IDOL for this challenge. In addition, we use pseudo labels to further help contrastive learning, so as to obtain more temporally consistent instance embedding to improve tracking performance between frames. The proposed method obtains 40.2 AP on the YouTube-VIS 2022 long video dataset and was ranked second place in this challenge. We hope our simple and effective method could benefit further research.

cs.CV↗

arXiv2022

Periodic Artifact Reduction in Fourier transforms of Full Field Atomic Resolution Images

The discrete Fourier transform is among the most routine tools used in high-resolution scanning / transmission electron microscopy (S/TEM). However, when calculating a Fourier transform, periodic boundary conditions are imposed and sharp discontinuities between the edges of an image cause a cross patterned artifact along the reciprocal space axes. This artifact can interfere with the analysis of reciprocal lattice peaks of an atomic resolution image. Here we demonstrate that the recently developed Periodic Plus Smooth Decomposition technique provides a simple, efficient method for reliable removal of artifacts caused by edge discontinuities. In this method, edge artifacts are reduced by subtracting a smooth background that solves Poisson's equation with boundary conditions set by the image's edges. Unlike the traditional windowed Fourier transforms, Periodic Plus Smooth Decomposition maintains sharp reciprocal lattice peaks from the image's entire field of view.

eess.IV↗

arXiv2022

Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoders

Masked autoencoders (MAEs) have emerged recently as art self-supervised spatiotemporal representation learners. Inheriting from the image counterparts, however, existing video MAEs still focus largely on static appearance learning whilst are limited in learning dynamic temporal information hence less effective for video downstream tasks. To resolve this drawback, in this work we present a motion-aware variant -- MotionMAE. Apart from learning to reconstruct individual masked patches of video frames, our model is designed to additionally predict the corresponding motion structure information over time. This motion information is available at the temporal difference of nearby frames. As a result, our model can extract effectively both static appearance and dynamic motion spontaneously, leading to superior spatiotemporal representation learning capability. Extensive experiments show that our MotionMAE outperforms significantly both supervised learning baseline and state-of-the-art MAE alternatives, under both domain-specific and domain-generic pretraining-then-finetuning settings. In particular, when using ViT-B as the backbone our MotionMAE surpasses the prior art model by a margin of 1.2% on Something-Something V2 and 3.2% on UCF101 in domain-specific pretraining setting. Encouragingly, it also surpasses the competing MAEs by a large margin of over 3% on the challenging video object segmentation task. The code is available at https://github.com/happy-hsy/MotionMAE.

cs.CV↗

arXiv2022

MechRetro is a chemical-mechanism-driven graph learning framework for interpretable retrosynthesis prediction and pathway planning

Leveraging artificial intelligence for automatic retrosynthesis speeds up organic pathway planning in digital laboratories. However, existing deep learning approaches are unexplainable, like "black box" with few insights, notably limiting their applications in real retrosynthesis scenarios. Here, we propose MechRetro, a chemical-mechanism-driven graph learning framework for interpretable retrosynthetic prediction and pathway planning, which learns several retrosynthetic actions to simulate a reverse reaction via elaborate self-adaptive joint learning. By integrating chemical knowledge as prior information, we design a novel Graph Transformer architecture to adaptively learn discriminative and chemically meaningful molecule representations, highlighting the strong capacity in molecule feature representation learning. We demonstrate that MechRetro outperforms the state-of-the-art approaches for retrosynthetic prediction with a large margin on large-scale benchmark datasets. Extending MechRetro to the multi-step retrosynthesis analysis, we identify efficient synthetic routes via an interpretable reasoning mechanism, leading to a better understanding in the realm of knowledgeable synthetic chemists. We also showcase that MechRetro discovers a novel pathway for protokylol, along with energy scores for uncertainty assessment, broadening the applicability for practical scenarios. Overall, we expect MechRetro to provide meaningful insights for high-throughput automated organic synthesis in drug discovery.

cs.LG↗

arXiv2022

Rethinking Resolution in the Context of Efficient Video Recognition

In this paper, we empirically study how to make the most of low-resolution frames for efficient video recognition. Existing methods mainly focus on developing compact networks or alleviating temporal redundancy of video inputs to increase efficiency, whereas compressing frame resolution has rarely been considered a promising solution. A major concern is the poor recognition accuracy on low-resolution frames. We thus start by analyzing the underlying causes of performance degradation on low-resolution frames. Our key finding is that the major cause of degradation is not information loss in the down-sampling process, but rather the mismatch between network architecture and input scale. Motivated by the success of knowledge distillation (KD), we propose to bridge the gap between network and input size via cross-resolution KD (ResKD). Our work shows that ResKD is a simple but effective method to boost recognition accuracy on low-resolution frames. Without bells and whistles, ResKD considerably surpasses all competitive methods in terms of efficiency and accuracy on four large-scale benchmark datasets, i.e., ActivityNet, FCVID, Mini-Kinetics, Something-Something V2. In addition, we extensively demonstrate its effectiveness over state-of-the-art architectures, i.e., 3D-CNNs and Video Transformers, and scalability towards super low-resolution frames. The results suggest ResKD can serve as a general inference acceleration method for state-of-the-art video recognition. Our code will be available at https://github.com/CVMI-Lab/ResKD.

cs.CV↗

arXiv2022

Deep learning at the edge enables real-time streaming ptychographic imaging

Coherent microscopy techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent X-ray microscopy methods like ptychography are poised to revolutionize nanoscale materials characterization. However, associated significant increases in data and compute needs mean that conventional approaches no longer suffice for recovering sample images in real-time from high-speed coherent imaging experiments. Here, we demonstrate a workflow that leverages artificial intelligence at the edge and high-performance computing to enable real-time inversion on X-ray ptychography data streamed directly from a detector at up to 2 kHz. The proposed AI-enabled workflow eliminates the sampling constraints imposed by traditional ptychography, allowing low dose imaging using orders of magnitude less data than required by traditional methods.

cs.LG↗

arXiv2022

Towards Grand Unification of Object Tracking

We present a unified method, termed Unicorn, that can simultaneously solve four tracking problems (SOT, MOT, VOS, MOTS) with a single network using the same model parameters. Due to the fragmented definitions of the object tracking problem itself, most existing trackers are developed to address a single or part of tasks and overspecialize on the characteristics of specific tasks. By contrast, Unicorn provides a unified solution, adopting the same input, backbone, embedding, and head across all tracking tasks. For the first time, we accomplish the great unification of the tracking network architecture and learning paradigm. Unicorn performs on-par or better than its task-specific counterparts in 8 tracking datasets, including LaSOT, TrackingNet, MOT17, BDD100K, DAVIS16-17, MOTS20, and BDD100K MOTS. We believe that Unicorn will serve as a solid step towards the general vision model. Code is available at https://github.com/MasterBin-IIAU/Unicorn.

cs.CV↗

arXiv2022

eeγproduction at photon-photon colliders at complete electroweak NLO accuracy

We present the NLO electroweak radiative corrections to the eeγproduction in γγcollision, which is an ideal channel for calibrating the beam luminosity of a Photon Linear Collider. We analyze the dependence of the total cross section on the beam colliding energy, and then investigate the kinematic distributions of final particles at various initial photon beam polarizations at EW NLO accuracy. The numerical results indicate that the EW relative corrections to the total cross section are non-negligible and become increasingly significant as the increase of the beam colliding energy. Such EW corrections are very important and should be taken into consideration in precision theoretical and experimental studies at high-energy γγcolliders.

hep-ph↗

arXiv2022

Real-time 3D analysis during electron tomography using tomviz

The demand for high-throughput electron tomography is rapidly increasing in biological and material sciences. However, this 3D imaging technique is computationally bottlenecked by alignment and reconstruction which runs from hours to days. We demonstrate real-time tomography with dynamic 3D tomographic visualization to enable rapid interpretation of specimen structure immediately as data is collected on an electron microscope. Using geometrically complex chiral nanoparticles, we show volumetric interpretation can begin in less than 10 minutes and a high quality tomogram is available within 30 minutes. Real time tomography is integrated into tomviz, an open source and cross platform 3D analysis tool that contains intuitive graphical user interfaces (GUI) to enable any scientist to characterize biological and material structure in 3D.

cond-mat.mtrl-sci↗

arXiv2022

Subtype-Former: a deep learning approach for cancer subtype discovery with multi-omics data

Motivation: Cancer is heterogeneous, affecting the precise approach to personalized treatment. Accurate subtyping can lead to better survival rates for cancer patients. High-throughput technologies provide multiple omics data for cancer subtyping. However, precise cancer subtyping remains challenging due to the large amount and high dimensionality of omics data. Results: This study proposed Subtype-Former, a deep learning method based on MLP and Transformer Block, to extract the low-dimensional representation of the multi-omics data. K-means and Consensus Clustering are also used to achieve accurate subtyping results. We compared Subtype-Former with the other state-of-the-art subtyping methods across the TCGA 10 cancer types. We found that Subtype-Former can perform better on the benchmark datasets of more than 5000 tumors based on the survival analysis. In addition, Subtype-Former also achieved outstanding results in pan-cancer subtyping, which can help analyze the commonalities and differences across various cancer types at the molecular level. Finally, we applied Subtype-Former to the TCGA 10 types of cancers. We identified 50 essential biomarkers, which can be used to study targeted cancer drugs and promote the development of cancer treatments in the era of precision medicine.

cs.LG↗