arXiv subjects

Peihan Miao

Publications and source records attributed to Peihan Miao.

7 recordsLinked to original sources

arXiv2024

ScanFormer: Referring Expression Comprehension by Iteratively Scanning

Referring Expression Comprehension (REC) aims to localize the target objects specified by free-form natural language descriptions in images. While state-of-the-art methods achieve impressive performance, they perform a dense perception of images, which incorporates redundant visual regions unrelated to linguistic queries, leading to additional computational overhead. This inspires us to explore a question: can we eliminate linguistic-irrelevant redundant visual regions to improve the efficiency of the model? Existing relevant methods primarily focus on fundamental visual tasks, with limited exploration in vision-language fields. To address this, we propose a coarse-to-fine iterative perception framework, called ScanFormer. It can iteratively exploit the image scale pyramid to extract linguistic-relevant visual patches from top to bottom. In each iteration, irrelevant patches are discarded by our designed informativeness prediction. Furthermore, we propose a patch selection strategy for discarded patches to accelerate inference. Experiments on widely used datasets, namely RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame, verify the effectiveness of our method, which can strike a balance between accuracy and efficiency.

cs.CV↗

arXiv2024

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

As an important and challenging problem in vision-language tasks, referring expression comprehension (REC) generally requires a large amount of multi-grained information of visual and linguistic modalities to realize accurate reasoning. In addition, due to the diversity of visual scenes and the variation of linguistic expressions, some hard examples have much more abundant multi-grained information than others. How to aggregate multi-grained information from different modalities and extract abundant knowledge from hard examples is crucial in the REC task. To address aforementioned challenges, in this paper, we propose a Self-paced Multi-grained Cross-modal Interaction Modeling framework, which improves the language-to-vision localization ability through innovations in network structure and learning mechanism. Concretely, we design a transformer-based multi-grained cross-modal attention, which effectively utilizes the inherent multi-grained information in visual and linguistic encoders. Furthermore, considering the large variance of samples, we propose a self-paced sample informativeness learning to adaptively enhance the network learning for samples containing abundant multi-grained information. The proposed framework significantly outperforms state-of-the-art methods on widely used datasets, such as RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame datasets, demonstrating the effectiveness of our method.

cs.CV↗

arXiv2023

Referring Expression Comprehension Using Language Adaptive Inference

Different from universal object detection, referring expression comprehension (REC) aims to locate specific objects referred to by natural language expressions. The expression provides high-level concepts of relevant visual and contextual patterns, which vary significantly with different expressions and account for only a few of those encoded in the REC model. This leads us to a question: do we really need the entire network with a fixed structure for various referring expressions? Ideally, given an expression, only expression-relevant components of the REC model are required. These components should be small in number as each expression only contains very few visual and contextual clues. This paper explores the adaptation between expressions and REC models for dynamic inference. Concretely, we propose a neat yet efficient framework named Language Adaptive Dynamic Subnets (LADS), which can extract language-adaptive subnets from the REC model conditioned on the referring expressions. By using the compact subnet, the inference can be more economical and efficient. Extensive experiments on RefCOCO, RefCOCO+, RefCOCOg, and Referit show that the proposed method achieves faster inference speed and higher accuracy against state-of-the-art approaches.

cs.CV↗

arXiv2023

Language Adaptive Weight Generation for Multi-task Visual Grounding

Although the impressive performance in visual grounding, the prevailing approaches usually exploit the visual backbone in a passive way, i.e., the visual backbone extracts features with fixed weights without expression-related hints. The passive perception may lead to mismatches (e.g., redundant and missing), limiting further performance improvement. Ideally, the visual backbone should actively extract visual features since the expressions already provide the blueprint of desired visual features. The active perception can take expressions as priors to extract relevant visual features, which can effectively alleviate the mismatches. Inspired by this, we propose an active perception Visual Grounding framework based on Language Adaptive Weights, called VG-LAW. The visual backbone serves as an expression-specific feature extractor through dynamic weights generated for various expressions. Benefiting from the specific and relevant visual features extracted from the language-aware visual backbone, VG-LAW does not require additional modules for cross-modal interaction. Along with a neat multi-task head, VG-LAW can be competent in referring expression comprehension and segmentation jointly. Extensive experiments on four representative datasets, i.e., RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame, validate the effectiveness of the proposed framework and demonstrate state-of-the-art performance.

cs.CV↗

arXiv2014

Nordhaus-Gaddum-type problems for lines in hypergraphs

We study the number of lines in hypergraphs in a more symmetric setting, where both the hypergraph and its complement are considered. In the general case and in some special cases, the lower bounds on the number of lines are much higher than their counterparts in single hypergraph setting or admit more elegant proofs. We show that the minimum value of product of the number of lines in both hypergraphs on $n$ points is easily determined as $\binom{n}{2}$; and the minimum value of their sum is between $Ω(n)$ and $O(n \log n)$. We also study some restricted classes of hypergraphs; and determine the tight bounds on the minimum sum when the hypergraph is derived from an Euclidean space, a real projective plane, or a tree.

math.CO↗

arXiv2014

Graph metric with no proper inclusion between lines

In trying to generalize the classic Sylvester-Gallai theorem and De Bruijn-Erdős theorem in plane geometry, lines and closure lines were previously defined for metric spaces and hypergraphs. Both definitions do not obey the geometric intuition in the sense that two lines (closure lines) may intersect at more than one point, and one line (closure line) might be the proper subset of another. In this work, we study the systems where one or both of the configurations are forbidden. We note that when any two lines intersect in at most one point, the two classic theorems extend in any metric space. We study the metric spaces induced by simple graphs where no line is a proper subset of another, and show that the least number of lines for such a graph with $n$ vertices is between the order of $n^{4/3}$ and $n^{4/3} \ln^{2/3} n$.

math.MG↗

arXiv2013

Number of lines in hypergraphs

Chen and Chvátal introduced the notion of lines in hypergraphs; they proved that every 3-uniform hypergraph with $n$ vertices either has a line that consists of all $n$ vertices or else has at least $\log_2 n$ distinct lines. We improve this lower bound by a factor of $2-o(1)$.

math.CO↗