arXiv subjects

Fang Peng

Publications and source records attributed to Fang Peng.

13 recordsLinked to original sources

arXiv2024

OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling

Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at https://github.com/linhuixiao/OneRef.

cs.CV↗

arXiv2024

SALI: Short-term Alignment and Long-term Interaction Network for Colonoscopy Video Polyp Segmentation

Colonoscopy videos provide richer information in polyp segmentation for rectal cancer diagnosis. However, the endoscope's fast moving and close-up observing make the current methods suffer from large spatial incoherence and continuous low-quality frames, and thus yield limited segmentation accuracy. In this context, we focus on robust video polyp segmentation by enhancing the adjacent feature consistency and rebuilding the reliable polyp representation. To achieve this goal, we in this paper propose SALI network, a hybrid of Short-term Alignment Module (SAM) and Long-term Interaction Module (LIM). The SAM learns spatial-aligned features of adjacent frames via deformable convolution and further harmonizes them to capture more stable short-term polyp representation. In case of low-quality frames, the LIM stores the historical polyp representations as a long-term memory bank, and explores the retrospective relations to interactively rebuild more reliable polyp features for the current segmentation. Combing SAM and LIM, the SALI network of video segmentation shows a great robustness to the spatial variations and low-visual cues. Benchmark on the large-scale SUNSEG verifies the superiority of SALI over the current state-of-the-arts by improving Dice by 2.1%, 2.5%, 4.1% and 1.9%, for the four test sub-sets, respectively. Codes are at https://github.com/Scatteredrain/SALI.

cs.CV↗

arXiv2024

HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding

Visual grounding, which aims to ground a visual region via natural language, is a task that heavily relies on cross-modal alignment. Existing works utilized uni-modal pre-trained models to transfer visual or linguistic knowledge separately while ignoring the multimodal corresponding information. Motivated by recent advancements in contrastive language-image pre-training and low-rank adaptation (LoRA) methods, we aim to solve the grounding task based on multimodal pre-training. However, there exists significant task gaps between pre-training and grounding. Therefore, to address these gaps, we propose a concise and efficient hierarchical multimodal fine-grained modulation framework, namely HiVG. Specifically, HiVG consists of a multi-layer adaptive cross-modal bridge and a hierarchical multimodal low-rank adaptation (HiLoRA) paradigm. The cross-modal bridge can address the inconsistency between visual features and those required for grounding, and establish a connection between multi-level visual and text features. HiLoRA prevents the accumulation of perceptual errors by adapting the cross-modal features from shallow to deep layers in a hierarchical manner. Experimental results on five datasets demonstrate the effectiveness of our approach and showcase the significant grounding capabilities as well as promising energy efficiency advantages. The project page: https://github.com/linhuixiao/HiVG.

cs.CV↗

arXiv2023

CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding

Visual Grounding (VG) is a crucial topic in the field of vision and language, which involves locating a specific region described by expressions within an image. To reduce the reliance on manually labeled data, unsupervised visual grounding have been developed to locate regions using pseudo-labels. However, the performance of existing unsupervised methods is highly dependent on the quality of pseudo-labels and these methods always encounter issues with limited diversity. In order to utilize vision and language pre-trained models to address the grounding problem, and reasonably take advantage of pseudo-labels, we propose CLIP-VG, a novel method that can conduct self-paced curriculum adapting of CLIP with pseudo-language labels. We propose a simple yet efficient end-to-end network architecture to realize the transfer of CLIP to the visual grounding. Based on the CLIP-based architecture, we further propose single-source and multi-source curriculum adapting algorithms, which can progressively find more reliable pseudo-labels to learn an optimal model, thereby achieving a balance between reliability and diversity for the pseudo-language labels. Our method outperforms the current state-of-the-art unsupervised method by a significant margin on RefCOCO/+/g datasets in both single-source and multi-source scenarios, with improvements ranging from 6.78$\%$ to 10.67$\%$ and 11.39$\%$ to 14.87$\%$, respectively. The results even outperform existing weakly supervised visual grounding methods. Furthermore, our method is also competitive in fully supervised setting. The code and models are available at https://github.com/linhuixiao/CLIP-VG.

cs.CV↗

arXiv2022

SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification

Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application. Recently, large-scale Vision-Language Pre-trained models (VLPs) have been gaining increasing attention in few-shot learning because they can provide a new paradigm for transferable visual representation learning with easily available text on the Web. However, the VLPs may neglect detailed visual information that is difficult to describe by language sentences, but important for learning an effective classifier to distinguish different images. To address the above problem, we propose a new framework, named Semantic-guided Visual Adapting (SgVA), which can effectively extend vision-language pre-trained models to produce discriminative adapted visual features by comprehensively using an implicit knowledge distillation, a vision-specific contrastive loss, and a cross-modal contrastive loss. The implicit knowledge distillation is designed to transfer the fine-grained cross-modal knowledge to guide the updating of the vision adapter. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.

cs.CV↗

arXiv2021

PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with \textit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$\alpha$, with up to 200 billion parameters. PanGu-$\alpha$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu-$\alpha$, we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu-$\alpha$ in various scenarios including text summarization, question answering, dialogue generation, etc. Moreover, we investigate the effect of model scales on the few-shot performances across a broad range of Chinese NLP tasks. The experimental results demonstrate the superior capabilities of PanGu-$\alpha$ in performing various tasks under few-shot or zero-shot settings.

cs.CL↗

arXiv2010

Helium Ignition on Accreting Neutron Stars with a New Triple-alpha Reaction Rate

We investigate the effect of a new triple-alpha reaction rate from Ogata et al. (2009) on helium ignition conditions on accreting neutron stars and on the properties of the subsequent type I X-ray burst. We find that the new rate leads to significantly lower ignition column density for accreting neutron stars at low accretion rates. We compare the results of our ignition models for a pure helium accretor to observations of bursts in ultra-compact X-ray binary (UCXBs), which are believed to have nearly pure helium donors. For mdot > 0.001 mdot_Edd, the new triple-alpha reaction rate from Ogata et al. (2009) predicts a maximum helium ignition column of ~ 3 x 10^9 g cm^{-2}, corresponding to a burst energy of ~ 4 x 10^{40} ergs. For mdot ~ 0.01 mdot_Edd at which intermediate long bursts occur, the predicted burst energies are at least a factor of 10 too low to explain the observed energies of such bursts in UCXBs. This finding adds to the doubts cast on the triple-alpha reaction rate of Ogata et al. (2009) by the low-mass stellar evolution results of Dotter & Paxton (2009).

astro-ph.HE↗

arXiv2009

Helium Sedimentation and the UV Upturn in Brightest Cluster Galaxies

Recent observations with Galaxy Evolution Explorer (GALEX) show strong unexpected UV excess in the spectrum of brightest cluster galaxies (BCGs). It is believed that the excess UV signal is produced by old and evolved core-He burning stars, and the UV flux strength could be greatly enhanced if the progenitor stars have high value of He abundance. In this work, we propose that sedimentation process can greatly enhance the He abundance in BCGs. Our model predicts that the UV flux strength is stronger in more massive, low-redshift, and dynamically relaxed BCGs. These predictions are testable with the current generation of GALEX+SDSS observations.

astro-ph.CO↗

arXiv2008

Effect of Helium Sedimentation on X-ray Measurements of Galaxy Clusters

The uniformity of the helium-to-hydrogen abundance ratio in X-ray emitting intracluster medium (ICM) is one of the commonly adopted assumptions in X-ray analyses of galaxy clusters and cosmological constraints derived from these measurements. In this work, we investigate the effect of He sedimentation on X-ray measurements of galaxy clusters in order to assess this assumption and associated systematic uncertainties. By solving a set of flow equations for a H-He plasma, we show that the helium-to-hydrogen mass ratio is significantly enhanced in the inner regions of clusters. The effect of He sedimentation, if not accounted for, introduces systematic biases in observable properties of clusters derived using X-ray observations. We show that these biases also introduce an apparent evolution in the observed gas mass fractions of X-ray luminous, dynamically relaxed clusters and hence biases in observational constraints on the dark energy equation of state parameter, w, derived from the cluster distance-redshift relation. The Hubble parameter derived from the combination of X-ray and Sunyaev-Zel'dovich effect (SZE) measurements is affected by the He sedimentation process as well. Future measurements aiming to constrain w or H_0 to better than 10% may need to take into account the effect of He sedimentation. We propose that the evolution of gas mass fraction in the inner regions of clusters should provide unique observational diagnostics of the He sedimentation process.

astro-ph↗

arXiv2006

Type I X-ray Bursts at Low Accretion Rates

Neutron stars, with their strong surface gravity, have interestingly short timescales for the sedimentation of heavy elements. Recent observations of unstable thermonuclear burning (observed as X-ray bursts) on the surfaces of slowly accreting neutron stars ($< 0.01$ of the Eddington rate) motivate us to examine how sedimentation of CNO isotopes affects the ignition of these bursts. We further estimate the burst development using a simple one-zone model with a full reaction network. We report a region of mass accretion rates for weak H flashes. Such flashes can lead to a large reservoir of He, the unstable burning of which may explain some observed long bursts (duration $\sim 1000$ s).

astro-ph↗

arXiv2006

Sedimentation and Type I X-ray Bursts at Low Accretion Rates

Neutron stars, with their strong surface gravity, have interestingly short timescales for the sedimentation of heavy elements. Motivated by observations of Type I X-ray bursts from sources with extremely low persistent accretion luminosities, $L_X < 10^{36}\usp\ergspersecond (\simeq 0.01\ensuremath{L_{\mathrm{Edd}}}$), we study how sedimentation affects the distribution of isotopes and the ignition of H and He in the envelope of an accreting neutron star. For local mass accretion rates $\mdot \lesssim 10^{-2}\medd$ (for which the ignition of H is unstable), where $\medd = 8.8\times 10^{4}\nsp\gpscps$, the helium and CNO elements sediment out of the accreted fuel before reaching a temperature where H would ignite. Using one-zone calculations of the thermonuclear burning, we find a range of accretion rates for which the unstable H ignition does not trigger unstable He burning. This range depends on the emergent flux from reactions in the deep neutron star crust; for $F = 0.1\nsp\MeV(\dot{m}/\mb)$, the range is $3\times 10^{-3}\medd\lesssim\mdot\lesssim 10^{-2}\medd$. We speculate that sources accreting in this range will build up a massive He layer that later produces an energetic and long X-ray burst. At mass accretion rates lower than this range, we find that the H flash leads to a strong mixed H/He flash. Surprisingly, even at accretion rates $\mdot \gtrsim 0.1\medd$, although the H and He do not completely segregate, the H abundance at the base of the accumulated layer is still reduced. While following the evolution of the X-ray burst is beyond the scope of this introductory paper, we note that the reduced proton-to-seed ratio favors the production of \iso{12}{C}--an important ingredient for subsequent superbursts.

astro-ph↗

arXiv2005

Two-Component Jet Models of Gamma-Ray Burst Sources

Recent observational and theoretical studies have raised the possibility that the collimated outflows in gamma-ray burst (GRB) sources have two distinct components: a narrow (opening half-angle $θ_{\rm n}$), highly relativistic (initial Lorentz factor $η_\rmn \gtrsim 10^2$) outflow, from which the $γ$-ray emission originates, and a wider ($θ_{\rm w} \lesssim 3 θ_{\rm n}$), moderately relativistic ($η_{\rm w}\sim 10$) surrounding flow. Using a simple synchrotron emission model, we calculate the R-band afterglow lightcurves expected in this scenario and derive algebraic expressions for the flux ratios of the emission from the two jet components at the main transition times in the lightcurve. We apply this model to GRB sources, for explaining the structure of afterglows and source energetics, as well as to X-ray flash sources, which we interpret as GRB jets viewed at an angle $θ_{\rm obs} > θ_{\rm n}$. Finally, we argue that a neutron-rich hydromagnetic outflow may naturally give rise to repeated brightening episodes in the afterglow lightcurve as observed in GRB 021004 and GRB 030329.

astro-ph↗

arXiv2003

Neutron-Rich Hydromagnetic Outflows in Gamma-Ray Burst Sources

We demonstrate that ``hot'' MHD outflows from neutron-rich black-hole debris disks can significantly alleviate the baryon-loading problem in gamma-ray burst (GRB) sources. We argue that the neutron-to-proton ratio in disk-fed outflows might be as high as ~30 and show, with the help of an exact semianalytic relativistic-MHD solution, that the neutrons can decouple at a Lorentz factor gamma_d ~ 15 even as the protons continue to accelerate to Lorentz factor ~200 and end up acquiring ~30% of the injected energy. We clarify the crucial role that the magnetic field plays in this process and prove that purely hydrodynamic outflows must have gamma_d > a few hundreds. The motion of the decoupled neutrons is not collinear with that of the decoupled protons, so, in contrast to previous suggestions based on purely hydrodynamic models, the two particle groups do not collide after decoupling. If the decoupled neutrons move at an angle > 1/gamma_d = 3.8 degr (15/gamma_d) to the line of sight to the GRB source, most of their emission after they decay into protons will remain unobservable.

astro-ph↗