Searcharxiv⌕ Search

arXiv subjects

Zixuan Zhu

Publications and source records attributed to Zixuan Zhu.

22 records · Page 2Linked to original sources

Bounding and Filling: A Fast and Flexible Framework for Image Captioning

Most image captioning models following an autoregressive manner suffer from significant inference latency. Several models adopted a non-autoregressive manner to speed up the process. However, the vanilla non-autoregressive manner results in subpar performance, since it generates all words simultaneously, which fails to capture the relationships between words in a description. The semi-autoregressive manner employs a partially parallel method to preserve performance, but it sacrifices inference speed. In this paper, we introduce a fast and flexible framework for image captioning called BoFiCap based on bounding and filling techniques. The BoFiCap model leverages the inherent characteristics of image captioning tasks to pre-define bounding boxes for image regions and their relationships. Subsequently, the BoFiCap model fills corresponding words in each box using two-generation manners. Leveraging the box hints, our filling process allows each word to better perceive other words. Additionally, our model offers flexible image description generation: 1) by employing different generation manners based on speed or performance requirements, 2) producing varied sentences based on user-specified boxes. Experimental evaluations on the MS-COCO benchmark dataset demonstrate that our framework in a non-autoregressive manner achieves the state-of-the-art on task-specific metric CIDEr (125.6) while speeding up 9.22x than the baseline model with an autoregressive manner; in a semi-autoregressive manner, our method reaches 128.4 on CIDEr while a 3.69x speedup. Our code and data is available at https://github.com/ChangxinWang/BoFiCap.

cs.CV↗

Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model

Current captioning approaches tend to generate correct but "generic" descriptions that lack real-world knowledge, e.g., named entities and contextual information. Considering that Vision-Language Pre-Training (VLP) models master massive such knowledge from large-scale web-harvested data, it is promising to utilize the generalizability of VLP models to incorporate knowledge into image descriptions. However, using VLP models faces challenges: zero-shot inference suffers from knowledge hallucination that leads to low-quality descriptions, but the generic bias in downstream task fine-tuning hinders the VLP model from expressing knowledge. To address these concerns, we propose a simple yet effective method called Knowledge-guided Replay (K-Replay), which enables the retention of pre-training knowledge during fine-tuning. Our approach consists of two parts: (1) a knowledge prediction task on automatically collected replay exemplars to continuously awaken the VLP model's memory about knowledge, thus preventing the model from collapsing into the generic pattern; (2) a knowledge distillation constraint to improve the faithfulness of generated descriptions hence alleviating the knowledge hallucination. To evaluate knowledge-enhanced descriptions, we construct a novel captioning benchmark KnowCap, containing knowledge of landmarks, famous brands, special foods and movie characters. Experimental results show that our approach effectively incorporates knowledge into descriptions, outperforming strong VLP baseline by 20.9 points (78.7->99.6) in CIDEr score and 20.5 percentage points (34.0%->54.5%) in knowledge recognition accuracy. Our code and data is available at https://github.com/njucckevin/KnowCap.

cs.CV↗

A Novel Scholar Embedding Model for Interdisciplinary Collaboration

Interdisciplinary collaboration has become a driving force for scientific breakthroughs, and evaluating scholars' performance in interdisciplinary researches is essential for promoting such collaborations. However, traditional scholar evaluation methods based solely on individual achievements do not consider interdisciplinary cooperation, creating a challenge for interdisciplinary scholar evaluation and recommendation. To address this issue, we propose a scholar embedding model that quantifies and represents scholars based on global semantic information and social influence, enabling real-time tracking of scholars' research trends. Our model incorporates semantic information and social influence for interdisciplinary scholar evaluation, laying the foundation for future interdisciplinary collaboration discovery and recommendation projects. We demonstrate the effectiveness of our model on a sample of scholars from the Beijing University of Posts and Telecommunications.

cs.DL↗

3D quench modeling based on T-A formulation for high temperature superconductor CORC cables

High temperature superconductor (HTS) (RE)Ba2Cu3Ox (REBCO) conductor on round core cable (CORC) has high current carrying capacity for high field magnet and power applications. In REBCO CORC cables, current redistribution occurs among tapes through terminal contact resistances when a local quench occurs. Therefore, the quench behaviour of CORC cable is different from single tape situation, for it is significantly affected by terminal contact resistances. To better understand the underlying physical process of local quenches in CORC cables, a new 3D multi-physics modelling tool for CORC cables is developed and presented in this paper. In this model, the REBCO tape is treated as a thin shell without thickness, and four models are coupled: T-formulation model, A-formulation model, a heat transfer model and an equivalent circuit model. The current redistribution, temperature and tape voltage of CORC cable during hot spot induced quenches are analysed using this model. The results show that the thermal stability of CORC cable can be considerably improved by reducing terminal contact resistance. The minimum quench energy (MQE) increases rapidly with the reduction of terminal contact resistance when the resistance is in a middle range. When the terminal contact resistance is too low or too high, the MQE shows no obvious variation with terminal contact resistances. With a low terminal contact resistance, a hot spot in one tape may induce an over-current quench on the other tapes without hot spots. This will not happen in a cable with high terminal contact resistance. In this case, the tape with hot spot will quench and burn out before inducing a quench on other tapes. The modelling tool developed can be used to design CORC cables with improved thermal stability.

cond-mat.supr-con↗