Searcharxiv⌕ Search

arXiv subjects

Shilong Zhang

Publications and source records attributed to Shilong Zhang.

At least 37 records · Page 2Linked to original sources

GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest

Visual instruction tuning large language model(LLM) on image-text pairs has achieved general-purpose vision-language abilities. However, the lack of region-text pairs limits their advancements to fine-grained multimodal understanding. In this paper, we propose spatial instruction tuning, which introduces the reference to the region-of-interest(RoI) in the instruction. Before sending to LLM, the reference is replaced by RoI features and interleaved with language embeddings as a sequence. Our model GPT4RoI, trained on 7 region-text pair datasets, brings an unprecedented interactive and conversational experience compared to previous image-level models. (1) Interaction beyond language: Users can interact with our model by both language and drawing bounding boxes to flexibly adjust the referring granularity. (2) Versatile multimodal abilities: A variety of attribute information within each RoI can be mined by GPT4RoI, e.g., color, shape, material, action, etc. Furthermore, it can reason about multiple RoIs based on common sense. On the Visual Commonsense Reasoning(VCR) dataset, GPT4RoI achieves a remarkable accuracy of 81.6%, surpassing all existing models by a significant margin (the second place is 75.6%) and almost reaching human-level performance of 85.0%. The code and model can be found at https://github.com/jshilong/GPT4RoI.

cs.CV↗

Dominant apical-oxygen electron-phonon coupling in HgBa$_2$Ca$_2$Cu$_3$O$_{8+δ}$

How electron-phonon interactions influence high-temperature superconductivity in cuprates remains contested, and their role outside the CuO$_2$ planes has been largely overlooked. The most conspicuous evidence for such coupling is the ubiquitous 70-meV dispersion kink seen by photoemission, yet its microscopic origin is still debated. Here we use oxygen-$K$-edge resonant inelastic X-ray scattering (RIXS) to probe the trilayer cuprate HgBa$_2$Ca$_2$Cu$_3$O$_{8+δ}$ (Hg1223). When both incident photon energy and polarization are tuned to the apical-oxygen $1s\!\rightarrow\!2p_z$ transition, the RIXS spectra exhibit a ladder of at least ten phonon overtones, evenly spaced by 70 meV, whose intensities follow a Franck-Condon envelope, signalling exceptionally strong electron-phonon coupling. Quantitative modelling that incorporates core-hole lifetime evaluation yields an apical-phonon coupling energy of 0.25(1) eV, significantly larger than that of the planar stretching mode. Such a coupling strength offers a strong contender for explaining the universal 70-meV kink and suggests that the dominant electron-phonon channel resides outside the CuO$_2$ planes. By elevating inter-layer lattice dynamics from a peripheral factor to a central actor, our results provide a fresh starting point for theories seeking to reconcile strong correlations, lattice dynamics and high-temperature superconductivity.

cond-mat.supr-con↗

NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images: Methods and Results

This paper reviews the NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images. This challenge received a wide range of impressive solutions, which are developed and evaluated using our collected real-world Raindrop Clarity dataset. Unlike existing deraining datasets, our Raindrop Clarity dataset is more diverse and challenging in degradation types and contents, which includes day raindrop-focused, day background-focused, night raindrop-focused, and night background-focused degradations. This dataset is divided into three subsets for competition: 14,139 images for training, 240 images for validation, and 731 images for testing. The primary objective of this challenge is to establish a new and powerful benchmark for the task of removing raindrops under varying lighting and focus conditions. There are a total of 361 participants in the competition, and 32 teams submitting valid solutions and fact sheets for the final testing phase. These submissions achieved state-of-the-art (SOTA) performance on the Raindrop Clarity dataset. The project can be found at https://lixinustc.github.io/CVPR-NTIRE2025-RainDrop-Competition.github.io/.

cs.CV↗

PixelFlow: Pixel-Space Generative Models with Flow

We present PixelFlow, a family of image generation models that operate directly in the raw pixel space, in contrast to the predominant latent-space models. This approach simplifies the image generation process by eliminating the need for a pre-trained Variational Autoencoder (VAE) and enabling the whole model end-to-end trainable. Through efficient cascade flow modeling, PixelFlow achieves affordable computation cost in pixel space. It achieves an FID of 1.98 on 256$\times$256 ImageNet class-conditional image generation benchmark. The qualitative text-to-image results demonstrate that PixelFlow excels in image quality, artistry, and semantic control. We hope this new paradigm will inspire and open up new opportunities for next-generation visual generation models. Code and models are available at https://github.com/ShoufaChen/PixelFlow.

cs.CV↗

Integro-differential rings on species and derived structures

In the theory of species, differential as well as integral operators are known to arise in a natural way. In this paper, we shall prove that they precisely fit together in the algebraic framework of integro-differential rings, which are themselves an abstraction of classical calculus (incorporating its Fundamental Theorem). The results comprise (set) species as well as linear species. Localization of (set) species leads to the more general structure of modified integro-differential rings, previously employed in the algebraic treatment of Volterra integral equations. Furthermore, the ring homomorphism from species to power series via taking generating series is shown to be a (modified) integro-differential ring homomorphism. As an application, a topology and further algebraic operations are imported to virtual species from the general theory of integro-differential rings.

math.CO↗

Goku: Flow Based Video Generative Foundation Models

This paper introduces Goku, a state-of-the-art family of joint image-and-video generation models leveraging rectified flow Transformers to achieve industry-leading performance. We detail the foundational elements enabling high-quality visual generation, including the data curation pipeline, model architecture design, flow formulation, and advanced infrastructure for efficient and robust large-scale training. The Goku models demonstrate superior performance in both qualitative and quantitative evaluations, setting new benchmarks across major tasks. Specifically, Goku achieves 0.76 on GenEval and 83.65 on DPG-Bench for text-to-image generation, and 84.85 on VBench for text-to-video tasks. We believe that this work provides valuable insights and practical advancements for the research community in developing joint image-and-video generation models.

cs.CV↗

Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM

Text-to-video models have made remarkable advancements through optimization on high-quality text-video pairs, where the textual prompts play a pivotal role in determining quality of output videos. However, achieving the desired output often entails multiple revisions and iterative inference to refine user-provided prompts. Current automatic methods for refining prompts encounter challenges such as Modality-Inconsistency, Cost-Discrepancy, and Model-Unaware when applied to text-to-video diffusion models. To address these problem, we introduce an LLM-based prompt adaptation framework, termed as Prompt-A-Video, which excels in crafting Video-Centric, Labor-Free and Preference-Aligned prompts tailored to specific video diffusion model. Our approach involves a meticulously crafted two-stage optimization and alignment system. Initially, we conduct a reward-guided prompt evolution pipeline to automatically create optimal prompts pool and leverage them for supervised fine-tuning (SFT) of the LLM. Then multi-dimensional rewards are employed to generate pairwise data for the SFT model, followed by the direct preference optimization (DPO) algorithm to further facilitate preference alignment. Through extensive experimentation and comparative analyses, we validate the effectiveness of Prompt-A-Video across diverse generation models, highlighting its potential to push the boundaries of video generation.

cs.CV↗

IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model

The rapid advancement of Large Vision-Language models (LVLMs) has demonstrated a spectrum of emergent capabilities. Nevertheless, current models only focus on the visual content of a single scenario, while their ability to associate instances across different scenes has not yet been explored, which is essential for understanding complex visual content, such as movies with multiple characters and intricate plots. Towards movie understanding, a critical initial step for LVLMs is to unleash the potential of character identities memory and recognition across multiple visual scenarios. To achieve the goal, we propose visual instruction tuning with ID reference and develop an ID-Aware Large Vision-Language Model, IDA-VLM. Furthermore, our research introduces a novel benchmark MM-ID, to examine LVLMs on instance IDs memory and recognition across four dimensions: matching, location, question-answering, and captioning. Our findings highlight the limitations of existing LVLMs in recognizing and associating instance identities with ID reference. This paper paves the way for future artificial intelligence systems to possess multi-identity visual inputs, thereby facilitating the comprehension of complex visual narratives like movies.

cs.CV↗

Zero-shot Image Editing with Reference Imitation

Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts is to precisely describe how the edited image should look like. In this work, we present a new form of editing, termed imitative editing, to help users exercise their creativity more conveniently. Concretely, to edit an image region of interest, users are free to directly draw inspiration from some in-the-wild references (e.g., some relative pictures come across online), without having to cope with the fit between the reference and the source. Such a design requires the system to automatically figure out what to expect from the reference to perform the editing. For this purpose, we propose a generative training framework, dubbed MimicBrush, which randomly selects two frames from a video clip, masks some regions of one frame, and learns to recover the masked regions using the information from the other frame. That way, our model, developed from a diffusion prior, is able to capture the semantic correspondence between separate images in a self-supervised manner. We experimentally show the effectiveness of our method under various test cases as well as its superiority over existing alternatives. We also construct a benchmark to facilitate further research.

cs.CV↗

Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

We introduce LlamaGen, a new family of image generation models that apply original ``next-token prediction'' paradigm of large language models to visual generation domain. It is an affirmative answer to whether vanilla autoregressive models, e.g., Llama, without inductive biases on visual signals can achieve state-of-the-art image generation performance if scaling properly. We reexamine design spaces of image tokenizers, scalability properties of image generation models, and their training data quality. The outcome of this exploration consists of: (1) An image tokenizer with downsample ratio of 16, reconstruction quality of 0.94 rFID and codebook usage of 97% on ImageNet benchmark. (2) A series of class-conditional image generation models ranging from 111M to 3.1B parameters, achieving 2.18 FID on ImageNet 256x256 benchmarks, outperforming the popular diffusion models such as LDM, DiT. (3) A text-conditional image generation model with 775M parameters, from two-stage training on LAION-COCO and high aesthetics quality images, demonstrating competitive performance of visual quality and text alignment. (4) We verify the effectiveness of LLM serving frameworks in optimizing the inference speed of image generation models and achieve 326% - 414% speedup. We release all models and codes to facilitate open-source community of visual generation and multimodal foundation models.

cs.CV↗

Observation of giant circular dichroism induced by electronic chirality

Chiral phases of matter, characterized by a definite handedness, abound in nature, ranging from the crystal structure of quartz to spiraling spin states in helical magnets. In $1T$-TiSe$_2$ a source of chirality has been proposed that stands apart from these classical examples as it arises from combined electronic charge and quantum orbital fluctuations. This may allow its chirality to be accessed and manipulated without imposing either structural or magnetic handedness. However, direct bulk evidence that broken inversion symmetry and chirality are intrinsic to TiSe$_2$ remains elusive. Here, employing resonant elastic scattering of x-rays, we reveal the presence of giant circular dichroism up to $\sim$ 40$\%$ at forbidden Bragg peaks that emerge at the charge and orbital ordering transition. The dichroism varies dramatically with incident energy and azimuthal angle. Comparison to calculated scattering intensities unambiguously traces its origin to bulk chiral electronic order in ${\mathrm{TiSe}}_2$ and establishes resonant elastic x-ray scattering as a sensitive probe to electronic chirality.

cond-mat.str-el↗

Charge redistribution, charge order and plasmon in La$_{2-x}$Sr$_{x}$CuO$_{4}$/La$_{2}$CuO$_{4}$ superlattices

Interfacial superconductors have the potential to revolutionize electronics, quantum computing, and fundamental physics due to their enhanced superconducting properties and ability to create new types of superconductors. The emergence of superconductivity at the interface of La$_{2-x}$Sr$_{x}$CuO$_{4}$/La$_{2}$CuO$_{4}$ (LSCO/LCO), with a T$_c$ enhancement of $\sim$ 10 K compared to the La$_{2-x}$Sr$_{x}$CuO$_{4}$ bulk single crystals, provides an exciting opportunity to study quantum phenomena in reduced dimensions. To investigate the carrier distribution and excitations in interfacial superconductors, we combine O K-edge resonant inelastic X-ray scattering and atomic-resolved scanning transmission electron microscopy measurements to study La$_{2-x}$Sr$_{x}$CuO$_{4}$/La$_{2}$CuO$_{4}$ superlattices (x=0.15, 0.45) and bulk La$_{1.55}$Sr$_{0.45}$CuO$_{4}$ films. We find direct evidence of charge redistribution, charge order and plasmon in LSCO/LCO superlattices. Notably, the observed behaviors of charge order and plasmon deviate from the anticipated properties of individual constituents or the average doping level of the superlattice. Instead, they conform harmoniously to the effective doping, a critical parameter governed by the T$_c$ of interfacial superconductors.

cond-mat.supr-con↗

Dense Distinct Query for End-to-End Object Detection

One-to-one label assignment in object detection has successfully obviated the need for non-maximum suppression (NMS) as postprocessing and makes the pipeline end-to-end. However, it triggers a new dilemma as the widely used sparse queries cannot guarantee a high recall, while dense queries inevitably bring more similar queries and encounter optimization difficulties. As both sparse and dense queries are problematic, then what are the expected queries in end-to-end object detection? This paper shows that the solution should be Dense Distinct Queries (DDQ). Concretely, we first lay dense queries like traditional detectors and then select distinct ones for one-to-one assignments. DDQ blends the advantages of traditional and recent end-to-end detectors and significantly improves the performance of various detectors including FCN, R-CNN, and DETRs. Most impressively, DDQ-DETR achieves 52.1 AP on MS-COCO dataset within 12 epochs using a ResNet-50 backbone, outperforming all existing detectors in the same setting. DDQ also shares the benefit of end-to-end detectors in crowded scenes and achieves 93.8 AP on CrowdHuman. We hope DDQ can inspire researchers to consider the complementarity between traditional methods and end-to-end detectors. The source code can be found at \url{https://github.com/jshilong/DDQ}.

cs.CV↗

MultiModal-GPT: A Vision and Language Model for Dialogue with Humans

We present a vision and language model named MultiModal-GPT to conduct multi-round dialogue with humans. MultiModal-GPT can follow various instructions from humans, such as generating a detailed caption, counting the number of interested objects, and answering general questions from users. MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with Low-rank Adapter (LoRA) added both in the cross-attention part and the self-attention part of the language model. We first construct instruction templates with vision and language data for multi-modality instruction tuning to make the model understand and follow human instructions. We find the quality of training data is vital for the dialogue performance, where few data containing short answers can lead the model to respond shortly to any instructions. To further enhance the ability to chat with humans of the MultiModal-GPT, we utilize language-only instruction-following data to train the MultiModal-GPT jointly. The joint training of language-only and visual-language instructions with the \emph{same} instruction template effectively improves dialogue performance. Various demos show the ability of continuous dialogue of MultiModal-GPT with humans. Code, dataset, and demo are at https://github.com/open-mmlab/Multimodal-GPT

cs.CV↗

Consistent-Teacher: Towards Reducing Inconsistent Pseudo-targets in Semi-supervised Object Detection

In this study, we dive deep into the inconsistency of pseudo targets in semi-supervised object detection (SSOD). Our core observation is that the oscillating pseudo-targets undermine the training of an accurate detector. It injects noise into the student's training, leading to severe overfitting problems. Therefore, we propose a systematic solution, termed ConsistentTeacher, to reduce the inconsistency. First, adaptive anchor assignment~(ASA) substitutes the static IoU-based strategy, which enables the student network to be resistant to noisy pseudo-bounding boxes. Then we calibrate the subtask predictions by designing a 3D feature alignment module~(FAM-3D). It allows each classification feature to adaptively query the optimal feature vector for the regression task at arbitrary scales and locations. Lastly, a Gaussian Mixture Model (GMM) dynamically revises the score threshold of pseudo-bboxes, which stabilizes the number of ground truths at an early stage and remedies the unreliable supervision signal during training. ConsistentTeacher provides strong results on a large range of SSOD evaluations. It achieves 40.0 mAP with ResNet-50 backbone given only 10% of annotated MS-COCO data, which surpasses previous baselines using pseudo labels by around 3 mAP. When trained on fully annotated MS-COCO with additional unlabeled data, the performance further increases to 47.7 mAP. Our code is available at \url{https://github.com/Adamdad/ConsistentTeacher}.

cs.CV↗

Coexistence of Multiple Stacking Charge Density Waves in Kagome Superconductor ${\mathrm{CsV}}_3{\mathrm{Sb}}_5$

The recently discovered Kagome family ${\mathrm{AV}}_3{\mathrm{Sb}}_5$ (A = K, Rb, Cs) exhibits rich physical phenomena, including non-trivial topological electronic structure, giant anomalous Hall effect, charge density waves (CDW) and superconductivity. Notably, CDW in ${\mathrm{AV}}_3{\mathrm{Sb}}_5$ is evidenced to intertwine with its superconductivity and topology, but its nature remains elusive. Here, we combine x-ray scattering experiments and density-functional theory calculations to investigate the CDWs in ${\mathrm{CsV}}_3{\mathrm{Sb}}_5$ and demonstrate the coexistence of 2 $\times$ 2 $\times$ 2 and 2 $\times$ 2 $\times$ 4 CDW stacking phases. Competition between these CDW phases is revealed by tracking the temperature evolution of CDW intensities, which also manifests in different transition temperatures during warming- and cooling measurements. We also identify a meta-stable quenched state of ${\mathrm{CsV}}_3{\mathrm{Sb}}_5$ after fast-cooling process. Our study demonstrates the coexistence of competing CDW stacking in ${\mathrm{CsV}}_3{\mathrm{Sb}}_5$, offering new insights in understanding the novel properties of this system.

cond-mat.supr-con↗

RTMDet: An Empirical Study of Designing Real-Time Object Detectors

In this paper, we aim to design an efficient real-time object detector that exceeds the YOLO series and is easily extensible for many object recognition tasks such as instance segmentation and rotated object detection. To obtain a more efficient model architecture, we explore an architecture that has compatible capacities in the backbone and neck, constructed by a basic building block that consists of large-kernel depth-wise convolutions. We further introduce soft labels when calculating matching costs in the dynamic label assignment to improve accuracy. Together with better training techniques, the resulting object detector, named RTMDet, achieves 52.8% AP on COCO with 300+ FPS on an NVIDIA 3090 GPU, outperforming the current mainstream industrial detectors. RTMDet achieves the best parameter-accuracy trade-off with tiny/small/medium/large/extra-large model sizes for various application scenarios, and obtains new state-of-the-art performance on real-time instance segmentation and rotated object detection. We hope the experimental results can provide new insights into designing versatile real-time object detectors for many object recognition tasks. Code and models are released at https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet.

cs.CV↗

Prevailing charge order in overdoped cuprates beyond the superconducting dome

The extremely overdoped cuprates are generally considered to be Fermi liquid metals without exotic orders, whereas the underdoped cuprates harbor intertwined states. Contrary to this conventional wisdom, using Cu $L_3$ edge and O $K$ edge resonant x-ray scattering, we reveal a charge order (CO) in overdoped La$_{2-x}$Sr$_x$CuO$_4$ (0.35 $\leq$ x $\leq$ 0.6) beyond the superconducting dome. This CO has a periodicity of $\sim$ 6 lattice units with correlation lengths of $\sim 3 - 20$ lattice units. It shows similar in-plane momentum and polarization dependence and dispersive excitations as the CO of underdoped cuprates, but its maximum intensity differs along the c-direction and persists up to 300 K. This CO cannot be explained by either the Fermi surface instability or the doped Hubbard model and its origin remains to be understood. Our results suggest that CO is prevailing in the overdoped metallic regime and superconductivity emerges out of the CO phase upon decreasing hole carriers.

cond-mat.supr-con↗