arXiv subjects

Chen Chen

Publications and source records attributed to Chen Chen.

At least 775 records · Page 43Linked to original sources

arXiv2022

M2HF: Multi-level Multi-modal Hybrid Fusion for Text-Video Retrieval

Videos contain multi-modal content, and exploring multi-level cross-modal interactions with natural language queries can provide great prominence to text-video retrieval task (TVR). However, new trending methods applying large-scale pre-trained model CLIP for TVR do not focus on multi-modal cues in videos. Furthermore, the traditional methods simply concatenating multi-modal features do not exploit fine-grained cross-modal information in videos. In this paper, we propose a multi-level multi-modal hybrid fusion (M2HF) network to explore comprehensive interactions between text queries and each modality content in videos. Specifically, M2HF first utilizes visual features extracted by CLIP to early fuse with audio and motion features extracted from videos, obtaining audio-visual fusion features and motion-visual fusion features respectively. Multi-modal alignment problem is also considered in this process. Then, visual features, audio-visual fusion features, motion-visual fusion features, and texts extracted from videos establish cross-modal relationships with caption queries in a multi-level way. Finally, the retrieval outputs from all levels are late fused to obtain final text-video retrieval results. Our framework provides two kinds of training strategies, including an ensemble manner and an end-to-end manner. Moreover, a novel multi-modal balance loss function is proposed to balance the contributions of each modality for efficient end-to-end training. M2HF allows us to obtain state-of-the-art results on various benchmarks, eg, Rank@1 of 64.9\%, 68.2\%, 33.2\%, 57.1\%, 57.8\% on MSR-VTT, MSVD, LSMDC, DiDeMo, and ActivityNet, respectively.

cs.MM↗

arXiv2022

Towards a category-extended object detector with limited data

Object detectors are typically learned on fully-annotated training data with fixed predefined categories. However, categories are often required to be increased progressively. Usually, only the original training set annotated with old classes and some new training data labeled with new classes are available in such scenarios. Based on the limited datasets, a unified detector that can handle all categories is strongly needed. We propose a practical scheme to achieve it in this work. A conflict-free loss is designed to avoid label ambiguity, leading to an acceptable detector in one training round. To further improve performance, we propose a retraining phase in which Monte Carlo Dropout is employed to calculate the localization confidence to mine more accurate bounding boxes, and an overlap-weighted method is proposed for making better use of pseudo annotations during retraining. Extensive experiments demonstrate the effectiveness of our method.

cs.CV↗

arXiv2022

Contact interaction analysis of octet baryon axialvector and pseudoscalar form factors

Octet baryon axial, induced pseudoscalar, and pseudoscalar form factors are computed using a symmetry-preserving treatment of a vector$\,\times\,$vector contact interaction (SCI), thereby unifying them with an array of other baryon properties and analogous treatments of semileptonic decays of pseudoscalar mesons. The baryons are treated as quark--plus--interacting-diquark bound states, whose structure is obtained by solving a Poincaré-covariant Faddeev equation. The approach is marked by algebraic simplicity, involves no free parameters, and since it is symmetry preserving, all consequences of partial conservation of the axial current are manifest. It is found that SCI results are consistent with only small violations of SU$(3)$-flavour symmetry, an outcome which may be understood as a dynamical consequence of emergent hadron mass. The spin-flavour structure of the Poincaré-covariant baryon wave functions is expressed in the presence of both flavour-antitriplet scalar diquarks and flavour-sextet axialvector diquarks and plays a key role in determining all form factors. Considering neutral axial currents, SCI predictions for the flavour separation of octet baryon axial charges and, therefrom, values for the associated SU$(3)$ singlet, triplet, and octet axial charges are obtained. The results indicate that at the hadron scale, $ζ_{\cal H}$, valence degrees-of-freedom carry roughly 50% of an octet baryon's total spin. Since there are no other degrees-of-freedom at $ζ_{\cal H}$, the remainder may be associated with quark+diquark orbital angular momentum.

hep-ph↗

arXiv2022

Joint Precoding and Phase Shift Design in Reconfigurable Intelligent Surfaces-Assisted Secret Key Generation

Key generation is a promising technique to establish symmetric keys between resource-constrained legitimate users. However, key generation suffers from low secret key rate (SKR) in harsh environments where channel randomness is limited. To address the problem, reconfigurable intelligent surfaces (RISs) are introduced to reshape the channels by controlling massive reflecting elements, which can provide more channel diversity. In this paper, we design a channel probing protocol to fully extract the randomness from the cascaded channel, i.e., the channel through reflecting elements. We derive the analytical expressions of SKR and design a water-filling algorithm based on the Karush-Kuhn-Tucker (KKT) conditions to find the upper bound. To find the optimal precoding and phase shift matrices, we propose an algorithm based on the Grassmann manifold optimization methods. The system is evaluated in terms of SKR, bit disagreement rate (BDR) and randomness. Simulation results show that our protocols significantly improve the SKR as compared to existing protocol.

cs.IT↗

arXiv2022

Towards Visualization of Time-Series Ecological Momentary Assessment (EMA) Data on Standalone Voice-First Virtual Assistants

Population aging is an increasingly important consideration for health care in the 21th century, and continuing to have access and interact with digital health information is a key challenge for aging populations. Voice-based Intelligent Virtual Assistants (IVAs) are promising to improve the Quality of Life (QoL) of older adults, and coupled with Ecological Momentary Assessments (EMA) they can be effective to collect important health information from older adults, especially when it comes to repeated time-based events. However, this same EMA data is hard to access for the older adult: although the newest IVAs are equipped with a display, the effectiveness of visualizing time-series based EMA data on standalone IVAs has not been explored. To investigate the potential opportunities for visualizing time-series based EMA data on standalone IVAs, we designed a prototype system, where older adults are able to query and examine the time-series EMA data on Amazon Echo Show - a widely used commercially available standalone screen-based IVA. We conducted a preliminary semi-structured interview with a geriatrician and an older adult, and identified three findings that should be carefully considered when designing such visualizations.

cs.HC↗

arXiv2022

Minimizing the programming power of phase change memory by using graphene nanoribbon edge-contact

Nonvolatile phase change random access memory (PCRAM) is regarded as one of promising candidates for emerging mass storage in the era of Big Data. However, relatively high programming energy hurdles the further reduction of power consumption in PCRAM. Utilizing narrow edge-contact of graphene can effectively reduce the active volume of phase change material in each cell, and therefore realize low-power operation. Here, we demonstrate that a write energy can be reduced to about ~53.7 fJ in a cell with ~3 nm-wide graphene nanoribbon (GNR) as edge-contact, whose cross-sectional area is only ~1 nm2. It is found that the cycle endurance exhibits an obvious dependence on the bias polarity in the cell with structure asymmetry. If a positive bias was applied to graphene electrode, the endurance can be extended at least one order longer than the case with reversal of polarity. The work represents a great technological advance for the low power PCRAM and could benefit for in-memory computing in future.

cond-mat.mes-hall↗

arXiv2022

Magic ELF: Image Deraining Meets Association Learning and Transformer

Convolutional neural network (CNN) and Transformer have achieved great success in multimedia applications. However, little effort has been made to effectively and efficiently harmonize these two architectures to satisfy image deraining. This paper aims to unify these two architectures to take advantage of their learning merits for image deraining. In particular, the local connectivity and translation equivariance of CNN and the global aggregation ability of self-attention (SA) in Transformer are fully exploited for specific local context and global structure representations. Based on the observation that rain distribution reveals the degradation location and degree, we introduce degradation prior to help background recovery and accordingly present the association refinement deraining scheme. A novel multi-input attention module (MAM) is proposed to associate rain perturbation removal and background recovery. Moreover, we equip our model with effective depth-wise separable convolutions to learn the specific feature representations and trade off computational complexity. Extensive experiments show that our proposed method (dubbed as ELF) outperforms the state-of-the-art approach (MPRNet) by 0.25 dB on average, but only accounts for 11.7\% and 42.1\% of its computational cost and parameters. The source code is available at https://github.com/kuijiang94/Magic-ELF.

cs.CV↗

arXiv2022

Estimating Model Performance under Domain Shifts with Class-Specific Confidence Scores

Machine learning models are typically deployed in a test setting that differs from the training setting, potentially leading to decreased model performance because of domain shift. If we could estimate the performance that a pre-trained model would achieve on data from a specific deployment setting, for example a certain clinic, we could judge whether the model could safely be deployed or if its performance degrades unacceptably on the specific data. Existing approaches estimate this based on the confidence of predictions made on unlabeled test data from the deployment's domain. We find existing methods struggle with data that present class imbalance, because the methods used to calibrate confidence do not account for bias induced by class imbalance, consequently failing to estimate class-wise accuracy. Here, we introduce class-wise calibration within the framework of performance estimation for imbalanced datasets. Specifically, we derive class-specific modifications of state-of-the-art confidence-based model evaluation methods including temperature scaling (TS), difference of confidences (DoC), and average thresholded confidence (ATC). We also extend the methods to estimate Dice similarity coefficient (DSC) in image segmentation. We conduct experiments on four tasks and find the proposed modifications consistently improve the estimation accuracy for imbalanced datasets. Our methods improve accuracy estimation by 18\% in classification under natural domain shifts, and double the estimation accuracy on segmentation tasks, when compared with prior methods.

cs.CV↗

arXiv2022

A Lightweight Graph Transformer Network for Human Mesh Reconstruction from 2D Human Pose

Existing deep learning-based human mesh reconstruction approaches have a tendency to build larger networks in order to achieve higher accuracy. Computational complexity and model size are often neglected, despite being key characteristics for practical use of human mesh reconstruction models (e.g. virtual try-on systems). In this paper, we present GTRS, a lightweight pose-based method that can reconstruct human mesh from 2D human pose. We propose a pose analysis module that uses graph transformers to exploit structured and implicit joint correlations, and a mesh regression module that combines the extracted pose feature with the mesh template to reconstruct the final human mesh. We demonstrate the efficiency and generalization of GTRS by extensive evaluations on the Human3.6M and 3DPW datasets. In particular, GTRS achieves better accuracy than the SOTA pose-based method Pose2Mesh while only using 10.2% of the parameters (Params) and 2.5% of the FLOPs on the challenging in-the-wild 3DPW dataset. Code will be publicly available.

cs.CV↗

arXiv2022

Masked Spatial-Spectral Autoencoders Are Excellent Hyperspectral Defenders

Deep learning methodology contributes a lot to the development of hyperspectral image (HSI) analysis community. However, it also makes HSI analysis systems vulnerable to adversarial attacks. To this end, we propose a masked spatial-spectral autoencoder (MSSA) in this paper under self-supervised learning theory, for enhancing the robustness of HSI analysis systems. First, a masked sequence attention learning module is conducted to promote the inherent robustness of HSI analysis systems along spectral channel. Then, we develop a graph convolutional network with learnable graph structure to establish global pixel-wise combinations.In this way, the attack effect would be dispersed by all the related pixels among each combination, and a better defense performance is achievable in spatial aspect.Finally, to improve the defense transferability and address the problem of limited labelled samples, MSSA employs spectra reconstruction as a pretext task and fits the datasets in a self-supervised manner.Comprehensive experiments over three benchmarks verify the effectiveness of MSSA in comparison with the state-of-the-art hyperspectral classification methods and representative adversarial defense strategies.

cs.CV↗

arXiv2022

GeoSegNet: Point Cloud Semantic Segmentation via Geometric Encoder-Decoder Modeling

Semantic segmentation of point clouds, aiming to assign each point a semantic category, is critical to 3D scene understanding.Despite of significant advances in recent years, most of existing methods still suffer from either the object-level misclassification or the boundary-level ambiguity. In this paper, we present a robust semantic segmentation network by deeply exploring the geometry of point clouds, dubbed GeoSegNet. Our GeoSegNet consists of a multi-geometry based encoder and a boundary-guided decoder. In the encoder, we develop a new residual geometry module from multi-geometry perspectives to extract object-level features. In the decoder, we introduce a contrastive boundary learning module to enhance the geometric representation of boundary points. Benefiting from the geometric encoder-decoder modeling, our GeoSegNet can infer the segmentation of objects effectively while making the intersections (boundaries) of two or more objects clear. Experiments show obvious improvements of our method over its competitors in terms of the overall segmentation accuracy and object boundary clearness. Code is available at https://github.com/Chen-yuiyui/GeoSegNet.

cs.CV↗

arXiv2022

Med-DANet: Dynamic Architecture Network for Efficient Medical Volumetric Segmentation

For 3D medical image (e.g. CT and MRI) segmentation, the difficulty of segmenting each slice in a clinical case varies greatly. Previous research on volumetric medical image segmentation in a slice-by-slice manner conventionally use the identical 2D deep neural network to segment all the slices of the same case, ignoring the data heterogeneity among image slices. In this paper, we focus on multi-modal 3D MRI brain tumor segmentation and propose a dynamic architecture network named Med-DANet based on adaptive model selection to achieve effective accuracy and efficiency trade-off. For each slice of the input 3D MRI volume, our proposed method learns a slice-specific decision by the Decision Network to dynamically select a suitable model from the predefined Model Bank for the subsequent 2D segmentation task. Extensive experimental results on both BraTS 2019 and 2020 datasets show that our proposed method achieves comparable or better results than previous state-of-the-art methods for 3D MRI brain tumor segmentation with much less model complexity. Compared with the state-of-the-art 3D method TransBTS, the proposed framework improves the model efficiency by up to 3.5x without sacrificing the accuracy. Our code will be publicly available soon.

eess.IV↗

arXiv2022

Experimental Observation and Description of Bandgaps Opening in Chiral Phononic Crystals by Analogy with Thomson scattering

Chiral phononic crystals provide unique properties not offered by conventional phononic material based on Bragg scattering and local resonance. However, it is insufficient to only consider the inertial amplification effect in chiral phononic crystals. Here, we theoretically and experimentally introduce the analogy with Thomson scattering to characterize the bandgap phenomena in chiral phononic crystals. Two phononic structures are proposed and discussed, one with translationrotation coupling and another with translation-translation coupling. The two lattices are different in appearance but have similar bandgaps. Thomson scattering in electromagnetic waves was drawn on to describe the coupling motion of the unit cells. We evidence that the bandgap generation is essentially based on the analogous Thomson scattering aiming to achieve an anti-phase superposition of the waves in the same polarization mode. This finding sheds new light on the physics of the elastodynamic wave manipulation in chiral phononic crystals and opens a remarkable route for their pragmatic implementation.

physics.app-ph↗

arXiv2022

GAMa: Cross-view Video Geo-localization

The existing work in cross-view geo-localization is based on images where a ground panorama is matched to an aerial image. In this work, we focus on ground videos instead of images which provides additional contextual cues which are important for this task. There are no existing datasets for this problem, therefore we propose GAMa dataset, a large-scale dataset with ground videos and corresponding aerial images. We also propose a novel approach to solve this problem. At clip-level, a short video clip is matched with corresponding aerial image and is later used to get video-level geo-localization of a long video. Moreover, we propose a hierarchical approach to further improve the clip-level geolocalization. It is a challenging dataset, unaligned and limited field of view, and our proposed method achieves a Top-1 recall rate of 19.4% and 45.1% @1.0mile. Code and dataset are available at following link: https://github.com/svyas23/GAMa.

cs.CV↗

arXiv2022

BANet: Motion Forecasting with Boundary Aware Network

We propose a motion forecasting model called BANet, which means Boundary-Aware Network, and it is a variant of LaneGCN. We believe that it is not enough to use only the lane centerline as input to obtain the embedding features of the vector map nodes. The lane centerline can only provide the topology of the lanes, and other elements of the vector map also contain rich information. For example, the lane boundary can provide traffic rule constraint information such as whether it is possible to change lanes which is very important. Therefore, we achieved better performance by encoding more vector map elements in the motion forecasting model.We report our results on the 2022 Argoverse2 Motion Forecasting challenge and rank 1st on the test leaderboard.

cs.CV↗

arXiv2022

Rethinking Reinforcement Learning based Logic Synthesis

Recently, reinforcement learning has been used to address logic synthesis by formulating the operator sequence optimization problem as a Markov decision process. However, through extensive experiments, we find out that the learned policy makes decisions independent from the circuit features (i.e., states) and yields an operator sequence that is permutation invariant to some extent in terms of operators. Based on these findings, we develop a new RL-based method that can automatically recognize critical operators and generate common operator sequences generalizable to unseen circuits. Our algorithm is verified on both the EPFL benchmark, a private dataset and a circuit at industrial scale. Experimental results demonstrate that it achieves a good balance among delay, area and runtime, and is practical for industrial usage.

cs.LG↗

arXiv2022

Nucleon axial form factor at large momentum transfers

Using a Poincaré-covariant quark+diquark Faddeev equation and related symmetry-preserving weak interaction current, we deliver parameter-free predictions for the nucleon axialvector form factor, $G_A(Q^2)$, on the domain $0\leq x=Q^2/m_N^2\leq 10$, where $m_N$ is the nucleon mass. We also provide a detailed analysis of the flavour separation of the proton $G_A$ into contributions from valence $u$ and $d$ quarks; and with form factors available on such a large $Q^2$ domain, predictions for the flavour-separated axial-charge light-front transverse spatial density profiles. Our calculated axial charge ratio $g_A^d/g_A^u=-0.32(2)$ is consistent with available experimental data and markedly larger in magnitude than the value typical of nonrelativistic quark models. The value of this ratio is sensitive to the strength of axialvector diquark correlations in the Poincaré-covariant nucleon wave function. Working with a realistic axialvector diquark content, the $d$ and $u$ quark transverse density profiles are similar. Some of these predictions could potentially be tested with new data on threshold pion electroproduction from the proton at large $Q^2$.

hep-ph↗

arXiv2022

Task-Adaptive Few-shot Node Classification

Node classification is of great importance among various graph mining tasks. In practice, real-world graphs generally follow the long-tail distribution, where a large number of classes only consist of limited labeled nodes. Although Graph Neural Networks (GNNs) have achieved significant improvements in node classification, their performance decreases substantially in such a few-shot scenario. The main reason can be attributed to the vast generalization gap between meta-training and meta-test due to the task variance caused by different node/class distributions in meta-tasks (i.e., node-level and class-level variance). Therefore, to effectively alleviate the impact of task variance, we propose a task-adaptive node classification framework under the few-shot learning setting. Specifically, we first accumulate meta-knowledge across classes with abundant labeled nodes. Then we transfer such knowledge to the classes with limited labeled nodes via our proposed task-adaptive modules. In particular, to accommodate the different node/class distributions among meta-tasks, we propose three essential modules to perform \emph{node-level}, \emph{class-level}, and \emph{task-level} adaptations in each meta-task, respectively. In this way, our framework can conduct adaptations to different meta-tasks and thus advance the model generalization performance on meta-test tasks. Extensive experiments on four prevalent node classification datasets demonstrate the superiority of our framework over the state-of-the-art baselines. Our code is provided at https://github.com/SongW-SW/TENT.

cs.LG↗