arXiv subjects

Pan Zhang

Publications and source records attributed to Pan Zhang.

At least 127 records · Page 7Linked to original sources

arXiv2023

V3Det: Vast Vocabulary Visual Detection Dataset

Recent advances in detecting arbitrary objects in the real world are trained and evaluated on object detection datasets with a relatively restricted vocabulary. To facilitate the development of more general visual object detection, we propose V3Det, a vast vocabulary visual detection dataset with precisely annotated bounding boxes on massive images. V3Det has several appealing properties: 1) Vast Vocabulary: It contains bounding boxes of objects from 13,204 categories on real-world images, which is 10 times larger than the existing large vocabulary object detection dataset, e.g., LVIS. 2) Hierarchical Category Organization: The vast vocabulary of V3Det is organized by a hierarchical category tree which annotates the inclusion relationship among categories, encouraging the exploration of category relationships in vast and open vocabulary object detection. 3) Rich Annotations: V3Det comprises precisely annotated objects in 243k images and professional descriptions of each category written by human experts and a powerful chatbot. By offering a vast exploration space, V3Det enables extensive benchmarks on both vast and open vocabulary object detection, leading to new observations, practices, and insights for future research. It has the potential to serve as a cornerstone dataset for developing more general visual perception systems. V3Det is available at https://v3det.openxlab.org.cn/.

cs.CV↗

arXiv2023

MLLM-DataEngine: An Iterative Refinement Approach for MLLM

Despite the great advance of Multimodal Large Language Models (MLLMs) in both instruction dataset building and benchmarking, the independence of training and evaluation makes current MLLMs hard to further improve their capability under the guidance of evaluation results with a relatively low human cost. In this paper, we propose MLLM-DataEngine, a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop iteration, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results, then generate a proper incremental dataset for the next training iteration and enhance the model capability iteratively. Compared with previous data collection methods which are separate from the benchmarking, the data generated by MLLM-DataEngine shows better targeting, quality, and correctness. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data within each incremental dataset based on the benchmarking results. For quality, we resort to GPT-4 to generate high-quality data with each given data type. For correctness, prompt design is critical for the data generation results. Rather than previous hand-crafted prompt, we propose an Interactive Prompt Optimization strategy, which optimizes the prompt with the multi-round interaction between human and GPT, and improve the correctness of generated data greatly. Through extensive experiments, we find our MLLM-DataEngine could boost the MLLM capability in a targeted and automatic manner, with only a few human participation. We hope it could be a general solution for the following MLLMs building. The MLLM-DataEngine has been open-sourced and is now available at https://github.com/opendatalab/MLLM-DataEngine.

cs.LG↗

arXiv2023

Fine-grained building roof instance segmentation based on domain adapted pretraining and composite dual-backbone

The diversity of building architecture styles of global cities situated on various landforms, the degraded optical imagery affected by clouds and shadows, and the significant inter-class imbalance of roof types pose challenges for designing a robust and accurate building roof instance segmentor. To address these issues, we propose an effective framework to fulfill semantic interpretation of individual buildings with high-resolution optical satellite imagery. Specifically, the leveraged domain adapted pretraining strategy and composite dual-backbone greatly facilitates the discriminative feature learning. Moreover, new data augmentation pipeline, stochastic weight averaging (SWA) training and instance segmentation based model ensemble in testing are utilized to acquire additional performance boost. Experiment results show that our approach ranks in the first place of the 2023 IEEE GRSS Data Fusion Contest (DFC) Track 1 test phase ($mAP_{50}$:50.6\%). Note-worthily, we have also explored the potential of multimodal data fusion with both optical satellite imagery and SAR data.

cs.CV↗

arXiv2023

HGDNet: A Height-Hierarchy Guided Dual-Decoder Network for Single View Building Extraction and Height Estimation

Unifying the correlative single-view satellite image building extraction and height estimation tasks indicates a promising way to share representations and acquire generalist model for large-scale urban 3D reconstruction. However, the common spatial misalignment between building footprints and stereo-reconstructed nDSM height labels incurs degraded performance on both tasks. To address this issue, we propose a Height-hierarchy Guided Dual-decoder Network (HGDNet) to estimate building height. Under the guidance of synthesized discrete height-hierarchy nDSM, auxiliary height-hierarchical building extraction branch enhance the height estimation branch with implicit constraints, yielding an accuracy improvement of more than 6% on the DFC 2023 track2 dataset. Additional two-stage cascade architecture is adopted to achieve more accurate building extraction. Experiments on the DFC 2023 Track 2 dataset shows the superiority of the proposed method in building height estimation (δ1:0.8012), instance extraction (AP50:0.7730), and the final average score 0.7871 ranks in the first place in test phase.

cs.CV↗

arXiv2023

Simulation of IBM's kicked Ising experiment with Projected Entangled Pair Operator

We perform classical simulations of the 127-qubit kicked Ising model, which was recently emulated using a quantum circuit with error mitigation [Nature 618, 500 (2023)]. Our approach is based on the projected entangled pair operator (PEPO) in the Heisenberg picture. Its main feature is the ability to automatically identify the underlying low-rank and low-entanglement structures in the quantum circuit involving Clifford and near-Clifford gates. We assess our approach using the quantum circuit with 5+1 trotter steps which was previously considered beyond classical verification. We develop a Clifford expansion theory to compute exact expectation values and use them to evaluate algorithms. The results indicate that PEPO significantly outperforms existing methods, including the tensor network with belief propagation, the matrix product operator, and the Clifford perturbation theory, in both efficiency and accuracy. In particular, PEPO with bond dimension $χ=2$ already gives similar accuracy to the CPT with $K=10$ and MPO with bond dimension $χ=1024$. And PEPO with $χ=184$ provides exact results in $3$ seconds using a single CPU. Furthermore, we apply our method to the circuit with 20 Trotter steps. We observe the monotonic and consistent convergence of the results with $χ$, allowing us to estimate the outcome with $χ\to\infty$ through extrapolations. We then compare the extrapolated results to those achieved in quantum hardware and with existing tensor network methods. Additionally, we discuss the potential usefulness of our approach in simulating quantum circuits, especially in scenarios involving near-Clifford circuits and quantum approximate optimization algorithms. Our approach is the first use of PEPO in solving the time evolution problem, and our results suggest it could be a powerful tool for exploring the dynamical properties of quantum many-body systems.

quant-ph↗

arXiv2023

qecGPT: decoding Quantum Error-correcting Codes with Generative Pre-trained Transformers

We propose a general framework for decoding quantum error-correcting codes with generative modeling. The model utilizes autoregressive neural networks, specifically Transformers, to learn the joint probability of logical operators and syndromes. This training is in an unsupervised way, without the need for labeled training data, and is thus referred to as pre-training. After the pre-training, the model can efficiently compute the likelihood of logical operators for any given syndrome, using maximum likelihood decoding. It can directly generate the most-likely logical operators with computational complexity $\mathcal O(2k)$ in the number of logical qubits $k$, which is significantly better than the conventional maximum likelihood decoding algorithms that require $\mathcal O(4^k)$ computation. Based on the pre-trained model, we further propose refinement to achieve more accurately the likelihood of logical operators for a given syndrome by directly sampling the stabilizer operators. We perform numerical experiments on stabilizer codes with small code distances, using both depolarizing error models and error models with correlated noise. The results show that our approach provides significantly better decoding accuracy than the minimum weight perfect matching and belief-propagation-based algorithms. Our framework is general and can be applied to any error model and quantum codes with different topologies such as surface codes and quantum LDPC codes. Furthermore, it leverages the parallelization capabilities of GPUs, enabling simultaneous decoding of a large number of syndromes. Our approach sheds light on the efficient and accurate decoding of quantum error-correcting codes using generative artificial intelligence and modern computational power.

quant-ph↗

arXiv2023

Tensor Network Message Passing

When studying interacting systems, computing their statistical properties is a fundamental problem in various fields such as physics, applied mathematics, and machine learning. However, this task can be quite challenging due to the exponential growth of the state space as the system size increases. Many standard methods have significant weaknesses. For instance, message-passing algorithms can be inaccurate and even fail to converge due to short loops. At the same time, tensor network methods can have exponential computational complexity in large graphs due to long loops. This work proposes a new method called ``tensor network message passing.'' This approach allows us to compute local observables like marginal probabilities and correlations by combining the strengths of tensor networks in contracting small sub-graphs with many short loops and the strengths of message-passing methods in globally sparse graphs, thus addressing the crucial weaknesses of both approaches. Our algorithm is exact for systems that are globally tree-like and locally dense-connected when the dense local graphs have limited treewidth. We have conducted numerical experiments on synthetic and real-world graphs to compute magnetizations of Ising models and spin glasses, to demonstrate the superiority of our approach over standard belief propagation and the recently proposed loopy message-passing algorithm. In addition, we discuss the potential applications of our method in inference problems in networks, combinatorial optimization problems, and decoding problems in quantum error correction.

cond-mat.stat-mech↗

arXiv2023

MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptation

In this work, we propose an ID-preserving talking head generation framework, which advances previous methods in two aspects. First, as opposed to interpolating from sparse flow, we claim that dense landmarks are crucial to achieving accurate geometry-aware flow fields. Second, inspired by face-swapping methods, we adaptively fuse the source identity during synthesis, so that the network better preserves the key characteristics of the image portrait. Although the proposed model surpasses prior generation fidelity on established benchmarks, to further make the talking head generation qualified for real usage, personalized fine-tuning is usually needed. However, this process is rather computationally demanding that is unaffordable to standard users. To solve this, we propose a fast adaptation model using a meta-learning approach. The learned model can be adapted to a high-quality personalized model as fast as 30 seconds. Last but not the least, a spatial-temporal enhancement module is proposed to improve the fine details while ensuring temporal coherency. Extensive experiments prove the significant superiority of our approach over the state of the arts in both one-shot and personalized settings.

cs.CV↗

arXiv2023

A Herculean task: Classical simulation of quantum computers

In the effort to develop useful quantum computers simulating quantum machines with conventional computing resources is a key capability. Such simulations will always face limits preventing the emulation of quantum computers of substantial scale but by pushing the envelope as far as possible through optimal choices of algorithms and hardware the value of the simulator tool is maximized. This work reviews the state-of-the-art numerical simulation methods i.e. the classical algorithms that emulate quantum computer evolution under specific operations. We focus on the mainstream state-vector and tensor-network paradigms while briefly mentioning alternative methods. Moreover we review the diverse applications of simulation across different facets of quantum computer development such as understanding the fundamental difference between quantum and classical computations exploring algorithm design spaces for quantum advantage predicting quantum processor performance at the design stage and characterizing fabricated devices efficiently for fast iterations. This review complements recent surveys on today's tools and implementations here we seek to acquaint the reader with an essential understanding of the theoretical basis of classical simulations detailed discussions on the advantages and limitations of different methods and the demands and challenges arising from practical use cases.

quant-ph↗

arXiv2023

Neural-network solutions to stochastic reaction networks

The stochastic reaction network in which chemical species evolve through a set of reactions is widely used to model stochastic processes in physics, chemistry and biology. To characterize the evolving joint probability distribution in the state space of species counts requires solving a system of ordinary differential equations, the chemical master equation, where the size of the counting state space increases exponentially with the type of species, making it challenging to investigate the stochastic reaction network. Here, we propose a machine-learning approach using the variational autoregressive network to solve the chemical master equation. Training the autoregressive network employs the policy gradient algorithm in the reinforcement learning framework, which does not require any data simulated in prior by another method. Different from simulating single trajectories, the approach tracks the time evolution of the joint probability distribution, and supports direct sampling of configurations and computing their normalized joint probabilities. We apply the approach to representative examples in physics and biology, and demonstrate that it accurately generates the probability distribution over time. The variational autoregressive network exhibits a plasticity in representing the multimodal distribution, cooperates with the conservation law, enables time-dependent reaction rates, and is efficient for high-dimensional reaction networks with allowing a flexible upper count limit. The results suggest a general approach to investigate stochastic reaction networks based on modern machine learning.

q-bio.MN↗

arXiv2023

Tensor networks for unsupervised machine learning

Modeling the joint distribution of high-dimensional data is a central task in unsupervised machine learning. In recent years, many interests have been attracted to developing learning models based on tensor networks, which have the advantages of a principle understanding of the expressive power using entanglement properties, and as a bridge connecting classical computation and quantum computation. Despite the great potential, however, existing tensor network models for unsupervised machine learning only work as a proof of principle, as their performance is much worse than the standard models such as restricted Boltzmann machines and neural networks. In this Letter, we present autoregressive matrix product states (AMPS), a tensor network model combining matrix product states from quantum many-body physics and autoregressive modeling from machine learning. Our model enjoys the exact calculation of normalized probability and unbiased sampling. We demonstrate the performance of our model using two applications, generative modeling on synthetic and real-world data, and reinforcement learning in statistical physics. Using extensive numerical experiments, we show that the proposed model significantly outperforms the existing tensor network models and the restricted Boltzmann machines, and is competitive with state-of-the-art neural network models.

cond-mat.stat-mech↗

arXiv2023

The Limit of the Yang-Mills-Higgs Flow for twisted Higgs pairs

In this paper, we consider the Yang-Mills-Higgs flow for twisted Higgs pairs over Kähler manifolds. We prove that this flow converges to a reflexive twisted Higgs sheaf outside a closed subset of codimension $4$, and the limiting twisted Higgs sheaf is isomorphic to the double dual of the graded twisted Higgs sheaves associated to the Harder-Narasimhan--eshadri filtration of the initial twisted Higgs bundle.

math.DG↗

arXiv2022

Origin of performance degradation in high-delithiation Li$_x$CoO$_2$: insights from direct atomic simulations using global neural network potentials

Li$_x$CoO$_2$ based batteries have serious capacity degradation and safety issues when cycling at high-delithiation states but full and consistent mechanisms are still poorly understood. Herein, a global neural network potential (GNNP) is developed to provide direct theoretical understandings by performing long-time and large-size atomic simulations. We propose a self-consistent picture as follows: (i) CoO$_2$ layers are easier to glide with longer distances at more highly delithiated states, resulting in structural transitions and structural inhomogeneity; (ii) at regions between different phases with different Li distributions due to gliding, local strains are induced and accumulate during cycling processes; (3) accumulated strains cause the rupture of Li diffusion channels and result in formation of oxygen dimers during cycling especially when Li has inhomogeneous distributions, leading to capacity degradations and safety issues. We find that large tensile strains combined with inhomogeneous distributions of Li ions play critical roles in the formation processes of blocked Li diffusion channels and the oxygen dimers at high-delithiation states, which could be the fundamental origins of capacity degradations and safety issues. Correspondingly, suppressing accumulations of strains by controlling charge and discharge conditions as well as suppressing the gliding will be helpful for improving the performance of lithium-ion batteries (LIBs).

physics.comp-ph↗

arXiv2022

Solving the sampling problem of the Sycamore quantum circuits

We study the problem of generating independent samples from the output distribution of Google's Sycamore quantum circuits with a target fidelity, which is believed to be beyond the reach of classical supercomputers and has been used to demonstrate quantum supremacy. We propose a new method to classically solve this problem by contracting the corresponding tensor network just once, and is massively more efficient than existing methods in obtaining a large number of uncorrelated samples with a target fidelity. For the Sycamore quantum supremacy circuit with $53$ qubits and $20$ cycles, we have generated one million uncorrelated bitstrings $\{\mathbf s\}$ which are sampled from a distribution $\hat P(\mathbf s)=|\hat ψ(\mathbf s)|^2$, where the approximate state $\hat ψ$ has fidelity $F\approx 0.0037$. The whole computation has cost about $15$ hours on a computational cluster with $512$ GPUs. The obtained one million samples, the contraction code and contraction order is made public. If our algorithm could be implemented with high efficiency on a modern supercomputer with ExaFLOPS performance, we estimate that ideally, the simulation would cost a few dozens of seconds, which is faster than Google's quantum hardware.

quant-ph↗

arXiv2022

A new higher order Yang--Mills--Higgs flow in Riemannian $4$-manifold

Let $(M,g)$ be a closed Riemannian $4$-manifold and let $E$ be a vector bundle over $M$ with structure group $G$, where $G$ is a compact Lie group. In this paper, we consider a new higher order Yang--Mills--Higgs functional, in which the Higgs field is a section of $Ω^0(\textmd{ad}E)$. We show that, under suitable conditions, solutions to the gradient flow do not hit any finite time singularities. In the case that $E$ is a line bundle, we are able to use a different blow up procedure and obtain an improvement of the long time result in \cite{Z1}. The proof is rather relevant to the properties of the Green function, which is very different from the previous techniques in \cite{Ke,Sa,Z1}.

math.DG↗

arXiv2022

Real-Time Neural Character Rendering with Pose-Guided Multiplane Images

We propose pose-guided multiplane image (MPI) synthesis which can render an animatable character in real scenes with photorealistic quality. We use a portable camera rig to capture the multi-view images along with the driving signal for the moving subject. Our method generalizes the image-to-image translation paradigm, which translates the human pose to a 3D scene representation -- MPIs that can be rendered in free viewpoints, using the multi-views captures as supervision. To fully cultivate the potential of MPI, we propose depth-adaptive MPI which can be learned using variable exposure images while being robust to inaccurate camera registration. Our method demonstrates advantageous novel-view synthesis quality over the state-of-the-art approaches for characters with challenging motions. Moreover, the proposed method is generalizable to novel combinations of training poses and can be explicitly controlled. Our method achieves such expressive and animatable character rendering all in real time, serving as a promising solution for practical applications.

cs.CV↗

arXiv2022

Semi-Supervised Image-to-Image Translation using Latent Space Mapping

Recent image-to-image translation works have been transferred from supervised to unsupervised settings due to the expensive cost of capturing or labeling large amounts of paired data. However, current unsupervised methods using the cycle-consistency constraint may not find the desired mapping, especially for difficult translation tasks. On the other hand, a small number of paired data are usually accessible. We therefore introduce a general framework for semi-supervised image translation. Unlike previous works, our main idea is to learn the translation over the latent feature space instead of the image space. Thanks to the low dimensional feature space, it is easier to find the desired mapping function, resulting in improved quality of translation results as well as the stability of the translation model. Empirically we show that using feature translation generates better results, even using a few bits of paired data. Experimental comparisons with state-of-the-art approaches demonstrate the effectiveness of the proposed framework on a variety of challenging image-to-image translation tasks

cs.CV↗

arXiv2021

Robust Mutual Learning for Semi-supervised Semantic Segmentation

Recent semi-supervised learning (SSL) methods are commonly based on pseudo labeling. Since the SSL performance is greatly influenced by the quality of pseudo labels, mutual learning has been proposed to effectively suppress the noises in the pseudo supervision. In this work, we propose robust mutual learning that improves the prior approach in two aspects. First, the vanilla mutual learners suffer from the coupling issue that models may converge to learn homogeneous knowledge. We resolve this issue by introducing mean teachers to generate mutual supervisions so that there is no direct interaction between the two students. We also show that strong data augmentations, model noises and heterogeneous network architectures are essential to alleviate the model coupling. Second, we notice that mutual learning fails to leverage the network's own ability for pseudo label refinement. Therefore, we introduce self-rectification that leverages the internal knowledge and explicitly rectifies the pseudo labels before the mutual teaching. Such self-rectification and mutual teaching collaboratively improve the pseudo label accuracy throughout the learning. The proposed robust mutual learning demonstrates state-of-the-art performance on semantic segmentation in low-data regime.

cs.CV↗