arXiv subjects

Cong Yao

Publications and source records attributed to Cong Yao.

At least 55 records · Page 3Linked to original sources

arXiv2022

Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion

Recently, segmentation-based scene text detection methods have drawn extensive attention in the scene text detection field, because of their superiority in detecting the text instances of arbitrary shapes and extreme aspect ratios, profiting from the pixel-level descriptions. However, the vast majority of the existing segmentation-based approaches are limited to their complex post-processing algorithms and the scale robustness of their segmentation models, where the post-processing algorithms are not only isolated to the model optimization but also time-consuming and the scale robustness is usually strengthened by fusing multi-scale feature maps directly. In this paper, we propose a Differentiable Binarization (DB) module that integrates the binarization process, one of the most important steps in the post-processing procedure, into a segmentation network. Optimized along with the proposed DB module, the segmentation network can produce more accurate results, which enhances the accuracy of text detection with a simple pipeline. Furthermore, an efficient Adaptive Scale Fusion (ASF) module is proposed to improve the scale robustness by fusing features of different scales adaptively. By incorporating the proposed DB and ASF with the segmentation network, our proposed scene text detector consistently achieves state-of-the-art results, in terms of both detection accuracy and speed, on five standard benchmarks.

cs.CV↗

arXiv2021

ARTS: Eliminating Inconsistency between Text Detection and Recognition with Auto-Rectification Text Spotter

Recent approaches for end-to-end text spotting have achieved promising results. However, most of the current spotters were plagued by the inconsistency problem between text detection and recognition. In this work, we introduce and prove the existence of the inconsistency problem and analyze it from two aspects: (1) inconsistency of text recognition features between training and testing, and (2) inconsistency of optimization targets between text detection and recognition. To solve the aforementioned issues, we propose a differentiable Auto-Rectification Module (ARM) together with a new training strategy to enable propagating recognition loss back into detection branch, so that our detection branch can be jointly optimized by detection and recognition targets, which largely alleviates the inconsistency problem between text detection and recognition. Based on these designs, we present a simple yet robust end-to-end text spotting framework, termed Auto-Rectification Text Spotter (ARTS), to detect and recognize arbitrarily-shaped text in natural scenes. Extensive experiments demonstrate the superiority of our method. In particular, our ARTS-S achieves 77.1% end-to-end text spotting F-measure on Total-Text at a competitive speed of 10.5 FPS, which significantly outperforms previous methods in both accuracy and inference speed.

cs.CV↗

arXiv2021

The Teichmüller problem for $L^p$-means of distortion

Teichmüller's problem from 1944 is this: Given $x\in [0,1)$ find and describe the extremal quasiconformal map $f:\ID\to\ID$, $f|\partial \ID=identity$ and $f(0)=-x\leq 0$. We consider this problem in the setting of minimisers of $L^p$-mean distortion. The classical result is that there is an extremal map of Teichmüller type with associated holomorphic quadratic differential having a pole of order one at $x$, if $x\neq 0$. For the $L^p$-norm, when $p=1$ it is known that there can be no locally quasiconformal minimiser unless $x=0$. Here we show that for $1\leq p<\infty$ there is a minimiser in a weak class and an associated Ahlfors-Hopf holomorphic quadratic differential with a pole of order $1$ at $f(0)=r$. However, this minimiser cannot be in $W^{1,2}_{loc}(\ID)$ unless $r=0$ and $f=identity$. Hence there is no locally quasiconformal minimiser. A similar statement holds for minimsers of the exponential norm of distortion. We also use our earlier work to show that as $p\to\infty$, the weak $L^p$-minimisers converge locally uniformly in $\ID$ to the extremal quasiconformal mapping, and that as $p\to 1$ the weak $L^p$-minimisers converge locally uniformly in $\ID$ to the identity.

math.CV↗

arXiv2021

Extremal mappings of finite distortion and the Radon-Riesz property

We consider Sobolev mappings $f\in W^{1,q}(Ω,\IC)$, $1<q<\infty$, between planar domains $Ω\subset \IC$. We analyse the Radon-Riesz property for convex functionals of the form \[f\mapsto \int_ΩΦ(|Df(z)|,J(z,f)) \; dz \] and show that under certain criteria, which hold in important cases, weak convergence in $W_{loc}^{1,q}(Ω)$ of (for instance) a minimising sequence can be improved to strong convergence. This finds important applications in the minimisation problems for mappings of finite distortion and the $L^p$ and $Exp$\,-Teichmüller theories.

math.CV↗

arXiv2021

Slender Object Detection: Diagnoses and Improvements

In this paper, we are concerned with the detection of a particular type of objects with extreme aspect ratios, namely \textbf{slender objects}. In real-world scenarios, slender objects are actually very common and crucial to the objective of a detection system. However, this type of objects has been largely overlooked by previous object detection algorithms. Upon our investigation, for a classical object detection method, a drastic drop of $18.9\%$ mAP on COCO is observed, if solely evaluated on slender objects. Therefore, we systematically study the problem of slender object detection in this work. Accordingly, an analytical framework with carefully designed benchmark and evaluation protocols is established, in which different algorithms and modules can be inspected and compared. \New Our study reveals that effective slender object detection can be achieved ~\textbf{with none of} (1) anchor-based localization; (2) specially designed box representations. Instead, \textbf{the critical aspect of improving slender object detection is feature adaptation}. It identifies and extends the insights of existing methods that are previously underexploited. Furthermore, we propose a feature adaption strategy that achieves clear and consistent improvements over current representative object detection methods.

cs.CV↗

arXiv2021

Facial Attribute Transformers for Precise and Robust Makeup Transfer

In this paper, we address the problem of makeup transfer, which aims at transplanting the makeup from the reference face to the source face while preserving the identity of the source. Existing makeup transfer methods have made notable progress in generating realistic makeup faces, but do not perform well in terms of color fidelity and spatial transformation. To tackle these issues, we propose a novel Facial Attribute Transformer (FAT) and its variant Spatial FAT for high-quality makeup transfer. Drawing inspirations from the Transformer in NLP, FAT is able to model the semantic correspondences and interactions between the source face and reference face, and then precisely estimate and transfer the facial attributes. To further facilitate shape deformation and transformation of facial parts, we also integrate thin plate splines (TPS) into FAT, thus creating Spatial FAT, which is the first method that can transfer geometric attributes in addition to color and texture. Extensive qualitative and quantitative experiments demonstrate the effectiveness and superiority of our proposed FATs in the following aspects: (1) ensuring high-fidelity color transfer; (2) allowing for geometric transformation of facial parts; (3) handling facial variations (such as poses and shadows) and (4) supporting high-resolution face generation.

cs.CV↗

arXiv2021

MOST: A Multi-Oriented Scene Text Detector with Localization Refinement

Over the past few years, the field of scene text detection has progressed rapidly that modern text detectors are able to hunt text in various challenging scenarios. However, they might still fall short when handling text instances of extreme aspect ratios and varying scales. To tackle such difficulties, we propose in this paper a new algorithm for scene text detection, which puts forward a set of strategies to significantly improve the quality of text localization. Specifically, a Text Feature Alignment Module (TFAM) is proposed to dynamically adjust the receptive fields of features based on initial raw detections; a Position-Aware Non-Maximum Suppression (PA-NMS) module is devised to selectively concentrate on reliable raw detections and exclude unreliable ones; besides, we propose an Instance-wise IoU loss for balanced training to deal with text instances of different scales. An extensive ablation study demonstrates the effectiveness and superiority of the proposed strategies. The resulting text detection system, which integrates the proposed strategies with a leading scene text detector EAST, achieves state-of-the-art or competitive performance on various standard benchmarks for text detection while keeping a fast running speed.

cs.CV↗

arXiv2020

TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes

Driven by deep neural networks and large scale datasets, scene text detection methods have progressed substantially over the past years, continuously refreshing the performance records on various standard benchmarks. However, limited by the representations (axis-aligned rectangles, rotated rectangles or quadrangles) adopted to describe text, existing methods may fall short when dealing with much more free-form text instances, such as curved text, which are actually very common in real-world scenarios. To tackle this problem, we propose a more flexible representation for scene text, termed as TextSnake, which is able to effectively represent text instances in horizontal, oriented and curved forms. In TextSnake, a text instance is described as a sequence of ordered, overlapping disks centered at symmetric axes, each of which is associated with potentially variable radius and orientation. Such geometry attributes are estimated via a Fully Convolutional Network (FCN) model. In experiments, the text detector based on TextSnake achieves state-of-the-art or comparable performance on Total-Text and SCUT-CTW1500, the two newly published benchmarks with special emphasis on curved text in natural images, as well as the widely-used datasets ICDAR 2015 and MSRA-TD500. Specifically, TextSnake outperforms the baseline on Total-Text by more than 40% in F-measure.

cs.CV↗

arXiv2020

UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World

Synthetic data has been a critical tool for training scene text detection and recognition models. On the one hand, synthetic word images have proven to be a successful substitute for real images in training scene text recognizers. On the other hand, however, scene text detectors still heavily rely on a large amount of manually annotated real-world images, which are expensive. In this paper, we introduce UnrealText, an efficient image synthesis method that renders realistic images via a 3D graphics engine. 3D synthetic engine provides realistic appearance by rendering scene and text as a whole, and allows for better text region proposals with access to precise scene information, e.g. normal and even object meshes. The comprehensive experiments verify its effectiveness on both scene text detection and recognition. We also generate a multilingual version for future research into multilingual scene text detection and recognition. Additionally, we re-annotate scene text recognition datasets in a case-sensitive way and include punctuation marks for more comprehensive evaluations. The code and the generated datasets are released at: https://github.com/Jyouhou/UnrealText/ .

cs.CV↗

arXiv2020

Scene Text Detection and Recognition: The Deep Learning Era

With the rise and development of deep learning, computer vision has been tremendously transformed and reshaped. As an important research area in computer vision, scene text detection and recognition has been inescapably influenced by this wave of revolution, consequentially entering the era of deep learning. In recent years, the community has witnessed substantial advancements in mindset, approach and performance. This survey is aimed at summarizing and analyzing the major changes and significant progresses of scene text detection and recognition in the deep learning era. Through this article, we devote to: (1) introduce new insights and ideas; (2) highlight recent techniques and benchmarks; (3) look ahead into future trends. Specifically, we will emphasize the dramatic differences brought by deep learning and the grand challenges still remained. We expect that this review paper would serve as a reference book for researchers in this field. Related resources are also collected and compiled in our Github repository: https://github.com/Jyouhou/SceneTextPapers.

cs.CV↗

arXiv2020

Differentiable Feature Aggregation Search for Knowledge Distillation

Knowledge distillation has become increasingly important in model compression. It boosts the performance of a miniaturized student network with the supervision of the output distribution and feature maps from a sophisticated teacher network. Some recent works introduce multi-teacher distillation to provide more supervision to the student network. However, the effectiveness of multi-teacher distillation methods are accompanied by costly computation resources. To tackle with both the efficiency and the effectiveness of knowledge distillation, we introduce the feature aggregation to imitate the multi-teacher distillation in the single-teacher distillation framework by extracting informative supervision from multiple teacher feature maps. Specifically, we introduce DFA, a two-stage Differentiable Feature Aggregation search method that motivated by DARTS in neural architecture search, to efficiently find the aggregations. In the first stage, DFA formulates the searching problem as a bi-level optimization and leverages a novel bridge loss, which consists of a student-to-teacher path and a teacher-to-student path, to find appropriate feature aggregations. The two paths act as two players against each other, trying to optimize the unified architecture parameters to the opposite directions while guaranteeing both expressivity and learnability of the feature aggregation simultaneously. In the second stage, DFA performs knowledge distillation with the derived feature aggregation. Experimental results show that DFA outperforms existing methods on CIFAR-100 and CINIC-10 datasets under various teacher-student settings, verifying the effectiveness and robustness of the design.

cs.LG↗

arXiv2020

The $L^p$ Teichmüller theory: Existence and regularity of critical points

We study minimisers of the $p$-conformal energy functionals, \[ \mathsf{E}_p(f):=\int_\ID \IK^p(z,f)\,dz,\quad f|_\IS=f_0|_\IS, \] defined for self mappings $f:\ID\to\ID$ with finite distortion and prescribed boundary values $f_0$. Here \[ \IK(z,f) = \frac{\|Df(z)\|^2}{J(z,f)} = \frac{1+|μ_f(z)|^2}{1-|μ_f(z)|^2}\] is the pointwise distortion functional and $μ_f(z)$ is the Beltrami coefficient of $f$. We show that for quasisymmetric boundary data the limiting regimes $p\to\infty$ recover the classical Teichmüller theory of extremal quasiconformal mappings (in part a result of Ahlfors), and for $p\to1$ recovers the harmonic mapping theory. Critical points of $\mathsf{E}_p$ always satisfy the inner-variational distributional equation \[ 2p\int_\ID \IK^p\;\frac{\overline{μ_f}}{1+|μ_f|^2}φ_\zbar \; dz=\int_\ID \IK^p \; φ_z\; dz,\quad\forallφ\in C_0^\infty(\ID ). \] We establish the existence of minimisers in the {\em a priori} regularity class $W^{1,\frac{2p}{p+1}}(\ID)$ and show these minimisers have a pseudo-inverse - a continuous $W^{1,2}(\ID)$ surjection of $\ID$ with $(h\circ f)(z)=z$ almost everywhere. We then give a sufficient condition to ensure $C^{\infty}(\ID)$ smoothness of solutions to the distributional equation. For instance $\IK(z,f)\in L^r_{loc}(\ID)$ for any $r>p+1$ is enough to imply the solutions to the distributional equation are local diffeomorphisms. Further $\IK(w,h)\in L^1(\ID)$ will imply $h$ is a homeomorphism, and together these results yield a diffeomorphic minimiser. We show such higher regularity assumptions to be necessary for critical points of the inner variational equation.

math.CV↗

arXiv2020

Higher regularity and uniqueness for inner variational equations

We study local minima of the $p$-conformal energy functionals, \[ \mathsf{E}_{\cal A}^\ast(h):=\int_\ID {\cal A}(\IK(w,h)) \;J(w,h) \; dw,\quad h|_\IS=h_0|_\IS, \] defined for self mappings $h:\ID\to\ID$ with finite distortion of the unit disk with prescribed boundary values $h_0$. Here $\IK(w,h) = \frac{\|Dh(w)\|^2}{J(w,h)} $ is the pointwise distortion functional, and ${\cal A}:[1,\infty)\to [1,\infty)$ is convex and increasing with ${\cal A}(t)\approx t^p$ for some $p\geq 1$, with additional minor technical conditions. Note ${\cal A}(t)=t$ is the Dirichlet energy functional. Critical points of $\mathsf{E}_{\cal A}^\ast$ satisfy the Ahlfors-Hopf inner-variational equation \[ {\cal A}'(\IK(w,h)) h_w \overline{h_\wbar} = Φ\] where $Φ$ is a holomorphic function. Iwaniec, Kovalev and Onninen established the Lipschitz regularity of critical points. Here we give a sufficient condition to ensure that a local minimum is a diffeomorphic solution to this equation, and that it is unique. This condition is necessarily satisfied by any locally quasiconformal critical point, and is basically the assumption $\IK(w,h)\in L^1(\ID)\cap L^r_{loc}(\ID)$ for some $r>1$.

math.CV↗

arXiv2020

On Vocabulary Reliance in Scene Text Recognition

The pursuit of high performance on public benchmarks has been the driving force for research in scene text recognition, and notable progress has been achieved. However, a close investigation reveals a startling fact that the state-of-the-art methods perform well on images with words within vocabulary but generalize poorly to images with words outside vocabulary. We call this phenomenon "vocabulary reliance". In this paper, we establish an analytical framework to conduct an in-depth study on the problem of vocabulary reliance in scene text recognition. Key findings include: (1) Vocabulary reliance is ubiquitous, i.e., all existing algorithms more or less exhibit such characteristic; (2) Attention-based decoders prove weak in generalizing to words outside vocabulary and segmentation-based decoders perform well in utilizing visual features; (3) Context modeling is highly coupled with the prediction layers. These findings provide new insights and can benefit future research in scene text recognition. Furthermore, we propose a simple yet effective mutual learning strategy to allow models of two families (attention-based and segmentation-based) to learn collaboratively. This remedy alleviates the problem of vocabulary reliance and improves the overall scene text recognition performance.

cs.CV↗

arXiv2020

A New Perspective for Flexible Feature Gathering in Scene Text Recognition Via Character Anchor Pooling

Irregular scene text recognition has attracted much attention from the research community, mainly due to the complexity of shapes of text in natural scene. However, recent methods either rely on shape-sensitive modules such as bounding box regression, or discard sequence learning. To tackle these issues, we propose a pair of coupling modules, termed as Character Anchoring Module (CAM) and Anchor Pooling Module (APM), to extract high-level semantics from two-dimensional space to form feature sequences. The proposed CAM localizes the text in a shape-insensitive way by design by anchoring characters individually. APM then interpolates and gathers features flexibly along the character anchors which enables sequence learning. The complementary modules realize a harmonic unification of spatial information and sequence learning. With the proposed modules, our recognition system surpasses previous state-of-the-art scores on irregular and perspective text datasets, including, ICDAR 2015, CUTE, and Total-Text, while paralleling state-of-the-art performance on regular text datasets.

cs.CV↗

arXiv2020

TextScanner: Reading Characters in Order for Robust Scene Text Recognition

Driven by deep learning and the large volume of data, scene text recognition has evolved rapidly in recent years. Formerly, RNN-attention based methods have dominated this field, but suffer from the problem of \textit{attention drift} in certain situations. Lately, semantic segmentation based algorithms have proven effective at recognizing text of different forms (horizontal, oriented and curved). However, these methods may produce spurious characters or miss genuine characters, as they rely heavily on a thresholding procedure operated on segmentation maps. To tackle these challenges, we propose in this paper an alternative approach, called TextScanner, for scene text recognition. TextScanner bears three characteristics: (1) Basically, it belongs to the semantic segmentation family, as it generates pixel-wise, multi-channel segmentation maps for character class, position and order; (2) Meanwhile, akin to RNN-attention based methods, it also adopts RNN for context modeling; (3) Moreover, it performs paralleled prediction for character position and class, and ensures that characters are transcripted in correct order. The experiments on standard benchmark datasets demonstrate that TextScanner outperforms the state-of-the-art methods. Moreover, TextScanner shows its superiority in recognizing more difficult text such Chinese transcripts and aligning with target characters.

cs.CV↗

arXiv2019

SynthText3D: Synthesizing Scene Text Images from 3D Virtual Worlds

With the development of deep neural networks, the demand for a significant amount of annotated training data becomes the performance bottlenecks in many fields of research and applications. Image synthesis can generate annotated images automatically and freely, which gains increasing attention recently. In this paper, we propose to synthesize scene text images from the 3D virtual worlds, where the precise descriptions of scenes, editable illumination/visibility, and realistic physics are provided. Different from the previous methods which paste the rendered text on static 2D images, our method can render the 3D virtual scene and text instances as an entirety. In this way, real-world variations, including complex perspective transformations, various illuminations, and occlusions, can be realized in our synthesized scene text images. Moreover, the same text instances with various viewpoints can be produced by randomly moving and rotating the virtual camera, which acts as human eyes. The experiments on the standard scene text detection benchmarks using the generated synthetic data demonstrate the effectiveness and superiority of the proposed method. The code and synthetic data is available at: https://github.com/MhLiao/SynthText3D

cs.CV↗

arXiv2019

Real-time Scene Text Detection with Differentiable Binarization

Recently, segmentation-based methods are quite popular in scene text detection, as the segmentation results can more accurately describe scene text of various shapes such as curve text. However, the post-processing of binarization is essential for segmentation-based detection, which converts probability maps produced by a segmentation method into bounding boxes/regions of text. In this paper, we propose a module named Differentiable Binarization (DB), which can perform the binarization process in a segmentation network. Optimized along with a DB module, a segmentation network can adaptively set the thresholds for binarization, which not only simplifies the post-processing but also enhances the performance of text detection. Based on a simple segmentation network, we validate the performance improvements of DB on five benchmark datasets, which consistently achieves state-of-the-art results, in terms of both detection accuracy and speed. In particular, with a light-weight backbone, the performance improvements by DB are significant so that we can look for an ideal tradeoff between detection accuracy and efficiency. Specifically, with a backbone of ResNet-18, our detector achieves an F-measure of 82.8, running at 62 FPS, on the MSRA-TD500 dataset. Code is available at: https://github.com/MhLiao/DB

cs.CV↗