Searcharxiv⌕ Search

arXiv subjects

Jihao Liu

Publications and source records attributed to Jihao Liu.

At least 91 records · Page 5Linked to original sources
arXiv2023

MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers

In this paper, we propose Mixed and Masked AutoEncoder (MixMAE), a simple but efficient pretraining method that is applicable to various hierarchical Vision Transformers. Existing masked image modeling (MIM) methods for hierarchical Vision Transformers replace a random subset of input tokens with a special [MASK] symbol and aim at reconstructing original image tokens from the corrupted image. However, we find that using the [MASK] symbol greatly slows down the training and causes pretraining-finetuning inconsistency, due to the large masking ratio (e.g., 60% in SimMIM). On the other hand, MAE does not introduce [MASK] tokens at its encoder at all but is not applicable for hierarchical Vision Transformers. To solve the issue and accelerate the pretraining of hierarchical models, we replace the masked tokens of one image with visible tokens of another image, i.e., creating a mixed image. We then conduct dual reconstruction to reconstruct the two original images from the mixed input, which significantly improves efficiency. While MixMAE can be applied to various hierarchical Transformers, this paper explores using Swin Transformer with a large window size and scales up to huge model size (to reach 600M parameters). Empirical results demonstrate that MixMAE can learn high-quality visual representations efficiently. Notably, MixMAE with Swin-B/W14 achieves 85.1% top-1 accuracy on ImageNet-1K by pretraining for 600 epochs. Besides, its transfer performances on the other 6 datasets show that MixMAE has better FLOPs / performance tradeoff than previous popular MIM methods. Code is available at https://github.com/Sense-X/MixMIM.

cs.CV↗
arXiv2023

On effective log Iitaka fibrations and existence of complements

We study the relationship between Iitaka fibrations and the conjecture on the existence of complements, assuming the good minimal model conjecture. In one direction, we show that the conjecture on the existence of complements implies the effective log Iitaka fibration conjecture. As a consequence, the effective log Iitaka fibration conjecture holds in dimension $3$. In the other direction, for any Calabi-Yau type variety $X$ such that $-K_X$ is nef, we show that $X$ has an $n$-complement for some universal constant $n$ depending only on the dimension of $X$ and two natural invariants of a general fiber of an Iitaka fibration of $-K_X$. We also formulate the decomposable Iitaka fibration conjecture, a variation of the effective log Iitaka fibration conjecture which is closely related to the structure of ample models of pairs with non-rational coefficients, and study its relationship with the forestated conjectures.

math.AG↗
arXiv2022

UniNet: Unified Architecture Search with Convolution, Transformer, and MLP

Recently, transformer and multi-layer perceptron (MLP) architectures have achieved impressive results on various vision tasks. However, how to effectively combine those operators to form high-performance hybrid visual architectures still remains a challenge. In this work, we study the learnable combination of convolution, transformer, and MLP by proposing a novel unified architecture search approach. Our approach contains two key designs to achieve the search for high-performance networks. First, we model the very different searchable operators in a unified form, and thus enable the operators to be characterized with the same set of configuration parameters. In this way, the overall search space size is significantly reduced, and the total search cost becomes affordable. Second, we propose context-aware downsampling modules (DSMs) to mitigate the gap between the different types of operators. Our proposed DSMs are able to better adapt features from different types of operators, which is important for identifying high-performance hybrid architectures. Finally, we integrate configurable operators and DSMs into a unified search space and search with a Reinforcement Learning-based search algorithm to fully explore the optimal combination of the operators. To this end, we search a baseline network and scale it up to obtain a family of models, named UniNets, which achieve much better accuracy and efficiency than previous ConvNets and Transformers. In particular, our UniNet-B5 achieves 84.9% top-1 accuracy on ImageNet, outperforming EfficientNet-B7 and BoTNet-T7 with 44% and 55% fewer FLOPs respectively. By pretraining on the ImageNet-21K, our UniNet-B6 achieves 87.4%, outperforming Swin-L with 51% fewer FLOPs and 41% fewer parameters. Code is available at https://github.com/Sense-X/UniNet.

cs.CV↗
arXiv2022

Uniform rational polytopes for Iitaka dimensions

In this paper, we continue to develop the theories on functional pairs and uniform rational polytopes. We show that there is a uniform perturbation for Iitaka dimensions of pseudo-effective lc pairs of fixed dimension with DCC coefficients assuming the non-vanishing conjecture. We also show the existence of uniform rational polytopes for Iitaka dimensions of pseudo-effective lc pairs assuming the non-vanishing conjecture.

math.AG↗
arXiv2022

INTERN: A New Learning Paradigm Towards General Vision

Enormous waves of technological innovations over the past several years, marked by the advances in AI technologies, are profoundly reshaping the industry and the society. However, down the road, a key challenge awaits us, that is, our capability of meeting rapidly-growing scenario-specific demands is severely limited by the cost of acquiring a commensurate amount of training data. This difficult situation is in essence due to limitations of the mainstream learning paradigm: we need to train a new model for each new scenario, based on a large quantity of well-annotated data and commonly from scratch. In tackling this fundamental problem, we move beyond and develop a new learning paradigm named INTERN. By learning with supervisory signals from multiple sources in multiple stages, the model being trained will develop strong generalizability. We evaluate our model on 26 well-known datasets that cover four categories of tasks in computer vision. In most cases, our models, adapted with only 10% of the training data in the target domain, outperform the counterparts trained with the full set of data, often by a significant margin. This is an important step towards a promising prospect where such a model with general vision capability can dramatically reduce our reliance on data, thus expediting the adoption of AI technologies. Furthermore, revolving around our new paradigm, we also introduce a new data system, a new architecture, and a new benchmark, which, together, form a general vision ecosystem to support its future development in an open and inclusive manner. See project website at https://opengvlab.shlab.org.cn .

cs.CV↗
arXiv2022

On generalized lc pairs with $\mathrm{\textbf b}$-log abundant nef part

We study the behavior of generalized lc pairs with $\mathrm{\textbf b}$-log abundant nef part, a meticulously designed structure on algebraic varieties. We show that this structure is preserved under the canonical bundle formula and sub-adjunction formulas, and is also compatible with the non-vanishing conjecture and the abundance conjecture in the classical minimal model program.

math.AG↗
arXiv2022

On the fixed part of pluricanonical systems for surfaces

We show that $|mK_X|$ defines a birational map and has no fixed part for some bounded positive integer $m$ for any $\frac{1}{2}$-lc surface $X$ such that $K_X$ is big and nef. For every positive integer $n\geq 3$, we construct a sequence of projective surfaces $X_{n,i}$, such that $K_{X_{n,i}}$ is ample, ${\rm{mld}}(X_{n,i})>\frac{1}{n}$ for every $i$, $\lim_{i\rightarrow+\infty}{\rm{mld}}(X_{n,i})=\frac{1}{n}$, and for any positive integer $m$, there exists $i$ such that $|mK_{X_{n,i}}|$ has non-zero fixed part. These results answer the surface case of a question of Xu.

math.AG↗
arXiv2022

Meta Knowledge Distillation

Recent studies pointed out that knowledge distillation (KD) suffers from two degradation problems, the teacher-student gap and the incompatibility with strong data augmentations, making it not applicable to training state-of-the-art models, which are trained with advanced augmentations. However, we observe that a key factor, i.e., the temperatures in the softmax functions for generating probabilities of both the teacher and student models, was mostly overlooked in previous methods. With properly tuned temperatures, such degradation problems of KD can be much mitigated. However, instead of relying on a naive grid search, which shows poor transferability, we propose Meta Knowledge Distillation (MKD) to meta-learn the distillation with learnable meta temperature parameters. The meta parameters are adaptively adjusted during training according to the gradients of the learning objective. We validate that MKD is robust to different dataset scales, different teacher/student architectures, and different types of data augmentation. With MKD, we achieve the best performance with popular ViT architectures among compared methods that use only ImageNet-1K as training data, ranging from tiny to large models. With ViT-L, we achieve 86.5% with 600 epochs of training, 0.6% better than MAE that trains for 1,650 epochs.

cs.LG↗
arXiv2022

ACC for minimal log discrepancies of terminal threefolds

We prove that the ACC conjecture for minimal log discrepancies holds for threefolds in $[1-δ,+\infty)$, where $δ>0$ only depends on the coefficient set. We also study Reid's general elephant for pairs, and show Shokurov's conjecture on the existence of $(ε,n)$-complements for threefolds for any $ε\geq 1$. As a key important step, we prove the uniform boundedness of divisors computing minimal log discrepancies for terminal threefolds. We show the ACC for threefold canonical thresholds, and that the set of accumulation points of threefold canonical thresholds is equal to $\{0\}\cup\{\frac{1}{n}\}_{n\in\mathbb Z_{\ge 2}}$ as well.

math.AG↗
arXiv2021

UniNet: Unified Architecture Search with Convolution, Transformer, and MLP

Recently, transformer and multi-layer perceptron (MLP) architectures have achieved impressive results on various vision tasks. A few works investigated manually combining those operators to design visual network architectures, and can achieve satisfactory performances to some extent. In this paper, we propose to jointly search the optimal combination of convolution, transformer, and MLP for building a series of all-operator network architectures with high performances on visual tasks. We empirically identify that the widely-used strided convolution or pooling based down-sampling modules become the performance bottlenecks when the operators are combined to form a network. To better tackle the global context captured by the transformer and MLP operators, we propose two novel context-aware down-sampling modules, which can better adapt to the global information encoded by transformer and MLP operators. To this end, we jointly search all operators and down-sampling modules in a unified search space. Notably, Our searched network UniNet (Unified Network) outperforms state-of-the-art pure convolution-based architecture, EfficientNet, and pure transformer-based architecture, Swin-Transformer, on multiple public visual benchmarks, ImageNet classification, COCO object detection, and ADE20K semantic segmentation.

cs.CV↗
arXiv2021

Existence of flips for generalized lc pairs

We prove the existence of flips for $\mathbb Q$-factorial NQC generalized lc pairs, and the cone and contraction theorems for NQC generalized lc pairs. This answers a question of C. Birkar which was conjectured by J. Han and Z. Li. As an immediate application, we show that we can run the minimal model program for $\mathbb Q$-factorial NQC generalized lc pairs. In particular, we complete the minimal model program for $\mathbb Q$-factorial NQC generalized lc pairs in dimension $\leq 3$ and pseudo-effective $\mathbb Q$-factorial NQC generalized lc pairs in dimension $4$.

math.AG↗
arXiv2021

FNAS: Uncertainty-Aware Fast Neural Architecture Search

Reinforcement learning (RL)-based neural architecture search (NAS) generally guarantees better convergence yet suffers from the requirement of huge computational resources compared with gradient-based approaches, due to the rollout bottleneck -- exhaustive training for each sampled generation on proxy tasks. In this paper, we propose a general pipeline to accelerate the convergence of the rollout process as well as the RL process in NAS. It is motivated by the interesting observation that both the architecture and the parameter knowledge can be transferred between different experiments and even different tasks. We first introduce an uncertainty-aware critic (value function) in Proximal Policy Optimization (PPO) to utilize the architecture knowledge in previous experiments, which stabilizes the training process and reduces the searching time by 4 times. Further, an architecture knowledge pool together with a block similarity function is proposed to utilize parameter knowledge and reduces the searching time by 2 times. It is the first to introduce block-level weight sharing in RLbased NAS. The block similarity function guarantees a 100% hitting ratio with strict fairness. Besides, we show that a simply designed off-policy correction factor used in "replay buffer" in RL optimization can further reduce half of the searching time. Experiments on the Mobile Neural Architecture Search (MNAS) search space show the proposed Fast Neural Architecture Search (FNAS) accelerates standard RL-based NAS process by ~10x (e.g. ~256 2x2 TPUv2 x days / 20,000 GPU x hour -> 2,000 GPU x hour for MNAS), and guarantees better performance on various vision tasks.

cs.LG↗
arXiv2021

Divisors computing minimal log discrepancies on lc surfaces

Let $(X\ni x,B)$ be an lc surface germ. If $X\ni x$ is klt, we show that there exists a divisor computing the minimal log discrepancy of $(X\ni x,B)$ that is a Kollár component of $X\ni x$. If $B\not=0$ or $X\ni x$ is not Du Val, we show that any divisor computing the minimal log discrepancy of $(X\ni x,B)$ is a potential lc place of $X\ni x$.

math.AG↗
arXiv2020

Learning Where to Focus for Efficient Video Object Detection

Transferring existing image-based detectors to the video is non-trivial since the quality of frames is always deteriorated by part occlusion, rare pose, and motion blur. Previous approaches exploit to propagate and aggregate features across video frames by using optical flow-warping. However, directly applying image-level optical flow onto the high-level features might not establish accurate spatial correspondences. Therefore, a novel module called Learnable Spatio-Temporal Sampling (LSTS) has been proposed to learn semantic-level correspondences among adjacent frame features accurately. The sampled locations are first randomly initialized, then updated iteratively to find better spatial correspondences guided by detection supervision progressively. Besides, Sparsely Recursive Feature Updating (SRFU) module and Dense Feature Aggregation (DFA) module are also introduced to model temporal relations and enhance per-frame features, respectively. Without bells and whistles, the proposed method achieves state-of-the-art performance on the ImageNet VID dataset with less computational complexity and real-time speed. Code will be made available at https://github.com/jiangzhengkai/LSTS.

cs.CV↗
arXiv2020

Effective birationality for sub-pairs with real coefficients

For $ε$-lc Fano type varieties $X$ of dimension $d$ and a given finite set $Γ$, we show that there exists a positive integer $m_0$ which only depends on $ε,d$ and $Γ$, such that both $|-mK_X-\sum_i\lceil mb_i\rceil B_i|$ and $|-mK_X-\sum_i\lfloor mb_i\rfloor B_i|$ define birational maps for any $m\ge m_0$ provided that $B_i$ are pseudo-effective Weil divisors, $b_i\inΓ$, and $-(K_X+\sum_ib_iB_i)$ is big. When $Γ\subset[0,1]$ satisfies the DCC but is not finite, we construct an example to show that the effective birationality may fail even if $X$ is fixed, $B_i$ are fixed prime divisors, and $(X,B)$ is $ε'$-lc for some $ε'>0$.

math.AG↗