arXiv subjects

Jun Yao

Publications and source records attributed to Jun Yao.

At least 37 records · Page 2Linked to original sources

arXiv2024

Multi-degree-of-freedom hybrid optical skyrmions

The optical counterparts of skyrmions have recently been constructed with diverse topological types and by different degrees of freedom, such as field, spins, and Stokes vectors, exhibiting extensive potential in modern information science. However, there is currently no method capable of generating multiple types of optical skyrmions in free space. Here, we present a simple approach for realizing hybrid optical skyrmions of electric field vectors, spin angular momentum and Stokes vectors in a same structured light field. We show that a vector beam truncated by an annular aperture can form an electric field skyrmion in the diffracted light field. In the meantime, electric field meron pairs, spin skyrmions and Stokes skyrmions can be generated by tuning spin-orbital coupling of the incident light.

physics.optics↗

arXiv2024

Compound Expression Recognition via Multi Model Ensemble for the ABAW7 Challenge

Compound Expression Recognition (CER) is vital for effective interpersonal interactions. Human emotional expressions are inherently complex due to the presence of compound expressions, requiring the consideration of both local and global facial cues for accurate judgment. In this paper, we propose an ensemble learning-based solution to address this complexity. Our approach involves training three distinct expression classification models using convolutional networks, Vision Transformers, and multiscale local attention networks. By employing late fusion for model ensemble, we combine the outputs of these models to predict the final results. Our method demonstrates high accuracy on the RAF-DB datasets and is capable of recognizing expressions in certain portions of the C-EXPR-DB through zero-shot learning.

cs.CV↗

arXiv2024

Facial Affect Recognition based on Multi Architecture Encoder and Feature Fusion for the ABAW7 Challenge

In this paper, we present our approach to addressing the challenges of the 7th ABAW competition. The competition comprises three sub-challenges: Valence Arousal (VA) estimation, Expression (Expr) classification, and Action Unit (AU) detection. To tackle these challenges, we employ state-of-the-art models to extract powerful visual features. Subsequently, a Transformer Encoder is utilized to integrate these features for the VA, Expr, and AU sub-challenges. To mitigate the impact of varying feature dimensions, we introduce an affine module to align the features to a common dimension. Overall, our results significantly outperform the baselines.

cs.CV↗

arXiv2024

MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time

Although Large Language Models (LLMs) achieve remarkable performance across various tasks, they often struggle with complex reasoning tasks, such as answering mathematical questions. Recent efforts to address this issue have primarily focused on leveraging mathematical datasets through supervised fine-tuning or self-improvement techniques. However, these methods often depend on high-quality datasets that are difficult to prepare, or they require substantial computational resources for fine-tuning. Inspired by findings that LLMs know how to produce the right answer but struggle to select the correct reasoning path, we propose a purely inference-based searching method -- MindStar (M*). This method formulates reasoning tasks as searching problems and proposes two search ideas to identify the optimal reasoning paths. We evaluate the M* framework on both the GSM8K and MATH datasets, comparing its performance with existing open and closed-source LLMs. Our results demonstrate that M* significantly enhances the reasoning abilities of open-source models, such as Llama-2-13B and Mistral-7B, and achieves comparable performance to GPT-3.5 and Grok-1, but with substantially reduced model size and computational costs.

cs.LG↗

arXiv2024

IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact

Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they compromise LLM performance. This paper unveils a previously overlooked type of outliers in LLMs. Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which are crucial to the performance of quantized LLMs. Given that, we propose IntactKV to generate the KV cache of pivot tokens losslessly from the full-precision model. The approach is simple and easy to combine with existing quantization solutions with no extra inference overhead. Besides, IntactKV can be calibrated as additional LLM parameters to boost the quantized LLMs further with minimal training costs. Mathematical analysis also proves that IntactKV effectively reduces the upper bound of quantization error. Empirical results show that IntactKV brings consistent improvement over various quantization methods across different LLMs and downstream tasks, leading to the new state-of-the-art for LLM quantization. The codes are available at https://github.com/ruikangliu/IntactKV.

cs.CL↗

arXiv2024

Basic (Dolbeault) Cohomology of Foliated Manifolds with Boundary

In this paper, we develop $L^2$ theory for Riemannian and Hermitian foliations on manifolds with basic boundary. We establish a decomposition theorem, various vanishing theorems, a twisted duality theorem for basic cohomologies and an extension theorem for basic forms of induced Riemannian foliation on the boundary. We prove the complex analogues for Hermitian foliations. To show the Dolbeault decomposition of basic forms, we extend Morrey's basic estimate to foliation version. We also investigate the global regularity for $\bar{\partial}_B$-equations.

math.DG↗

arXiv2024

Machine Learning Insides OptVerse AI Solver: Design Principles and Applications

In an era of digital ubiquity, efficient resource management and decision-making are paramount across numerous industries. To this end, we present a comprehensive study on the integration of machine learning (ML) techniques into Huawei Cloud's OptVerse AI Solver, which aims to mitigate the scarcity of real-world mathematical programming instances, and to surpass the capabilities of traditional optimization techniques. We showcase our methods for generating complex SAT and MILP instances utilizing generative models that mirror multifaceted structures of real-world problem. Furthermore, we introduce a training framework leveraging augmentation policies to maintain solvers' utility in dynamic environments. Besides the data generation and augmentation, our proposed approaches also include novel ML-driven policies for personalized solver strategies, with an emphasis on applications like graph convolutional networks for initial basis selection and reinforcement learning for advanced presolving and cut selection. Additionally, we detail the incorporation of state-of-the-art parameter tuning algorithms which markedly elevate solver performance. Compared with traditional solvers such as Cplex and SCIP, our ML-augmented OptVerse AI Solver demonstrates superior speed and precision across both established benchmarks and real-world scenarios, reinforcing the practical imperative and effectiveness of machine learning techniques in mathematical programming solvers.

cs.AI↗

arXiv2023

PanGu-$π$: Enhancing Language Model Architectures via Nonlinearity Compensation

The recent trend of large language models (LLMs) is to increase the scale of both model size (\aka the number of parameters) and dataset to achieve better generative ability, which is definitely proved by a lot of work such as the famous GPT and Llama. However, large models often involve massive computational costs, and practical applications cannot afford such high prices. However, the method of constructing a strong model architecture for LLMs is rarely discussed. We first analyze the state-of-the-art language model architectures and observe the feature collapse problem. Based on the theoretical analysis, we propose that the nonlinearity is also very important for language models, which is usually studied in convolutional neural networks for vision tasks. The series informed activation function is then introduced with tiny calculations that can be ignored, and an augmented shortcut is further used to enhance the model nonlinearity. We then demonstrate that the proposed approach is significantly effective for enhancing the model nonlinearity through carefully designed ablations; thus, we present a new efficient model architecture for establishing modern, namely, PanGu-$π$. Experiments are then conducted using the same dataset and training strategy to compare PanGu-$π$ with state-of-the-art LLMs. The results show that PanGu-$π$-7B can achieve a comparable performance to that of benchmarks with about 10\% inference speed-up, and PanGu-$π$-1B can achieve state-of-the-art performance in terms of accuracy and efficiency. In addition, we have deployed PanGu-$π$-7B in the high-value domains of finance and law, developing an LLM named YunShan for practical application. The results show that YunShan can surpass other models with similar scales on benchmarks.

cs.CL↗

arXiv2023

ArCL: Enhancing Contrastive Learning with Augmentation-Robust Representations

Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data for model training. Empirical studies show that SSL can achieve promising performance in distribution shift scenarios, where the downstream and training distributions differ. However, the theoretical understanding of its transferability remains limited. In this paper, we develop a theoretical framework to analyze the transferability of self-supervised contrastive learning, by investigating the impact of data augmentation on it. Our results reveal that the downstream performance of contrastive learning depends largely on the choice of data augmentation. Moreover, we show that contrastive learning fails to learn domain-invariant features, which limits its transferability. Based on these theoretical insights, we propose a novel method called Augmentation-robust Contrastive Learning (ArCL), which guarantees to learn domain-invariant features and can be easily integrated with existing contrastive learning algorithms. We conduct experiments on several datasets and show that ArCL significantly improves the transferability of contrastive learning.

cs.LG↗

arXiv2023

When Noisy Labels Meet Long Tail Dilemmas: A Representation Calibration Method

Real-world large-scale datasets are both noisily labeled and class-imbalanced. The issues seriously hurt the generalization of trained models. It is hence significant to address the simultaneous incorrect labeling and class-imbalance, i.e., the problem of learning with noisy labels on long-tailed data. Previous works develop several methods for the problem. However, they always rely on strong assumptions that are invalid or hard to be checked in practice. In this paper, to handle the problem and address the limitations of prior works, we propose a representation calibration method RCAL. Specifically, RCAL works with the representations extracted by unsupervised contrastive learning. We assume that without incorrect labeling and class imbalance, the representations of instances in each class conform to a multivariate Gaussian distribution, which is much milder and easier to be checked. Based on the assumption, we recover underlying representation distributions from polluted ones resulting from mislabeled and class-imbalanced data. Additional data points are then sampled from the recovered distributions to help generalization. Moreover, during classifier training, representation learning takes advantage of representation robustness brought by contrastive learning, which further improves the classifier performance. We derive theoretical results to discuss the effectiveness of our representation calibration. Experiments on multiple benchmarks justify our claims and confirm the superiority of the proposed method.

cs.LG↗

arXiv2023

PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1.085T parameters named PanGu-Σ. With parameter inherent from PanGu-α, we extend the dense Transformer model to sparse one with Random Routed Experts (RRE), and efficiently train the model over 329B tokens by using Expert Computation and Storage Separation(ECSS). This resulted in a 6.3x increase in training throughput through heterogeneous computing. Our experimental findings show that PanGu-Σ provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks. Moreover, it demonstrates strong abilities when fine-tuned in application data of open-domain dialogue, question answering, machine translation and code generation.

cs.CL↗

arXiv2022

Generation of optical vortices imitating water vortices

In optics, we can generate vortex beams using specific methods such as spiral phase plates or computer generated holograms. While, in nature, it is worth noting that water can produce vortices by a circularly symmetrical hole. So, if a light beam can generate vortex when it is diffracted by an aperture? Here, we show that the light field in the Fresnel region of the diffracted circularly polarized beam carries orbital angular momentum, which can transfer to the trapped particles and make orbital rotation.

physics.optics↗

arXiv2021

Sub-Architecture Ensemble Pruning in Neural Architecture Search

Neural architecture search (NAS) is gaining more and more attention in recent years due to its flexibility and remarkable capability to reduce the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computations that might not be affordable for researchers and practitioners. While recent attempts have employed ensemble learning methods to mitigate the enormous computational cost, however, they neglect a key property of ensemble methods, namely diversity, which leads to collecting more similar sub-architectures with potential redundancy in the final design. To tackle this problem, we propose a pruning method for NAS ensembles called "Sub-Architecture Ensemble Pruning in Neural Architecture Search (SAEP)." It targets to leverage diversity and to achieve sub-ensemble architectures at a smaller size with comparable performance to ensemble architectures that are not pruned. Three possible solutions are proposed to decide which sub-architectures to prune during the searching process. Experimental results exhibit the effectiveness of the proposed method by largely reducing the number of sub-architectures without degrading the performance.

cs.LG↗

arXiv2021

Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution

Portfolio management via reinforcement learning is at the forefront of fintech research, which explores how to optimally reallocate a fund into different financial assets over the long term by trial-and-error. Existing methods are impractical since they usually assume each reallocation can be finished immediately and thus ignoring the price slippage as part of the trading cost. To address these issues, we propose a hierarchical reinforced stock trading system for portfolio management (HRPM). Concretely, we decompose the trading process into a hierarchy of portfolio management over trade execution and train the corresponding policies. The high-level policy gives portfolio weights at a lower frequency to maximize the long term profit and invokes the low-level policy to sell or buy the corresponding shares within a short time window at a higher frequency to minimize the trading cost. We train two levels of policies via pre-training scheme and iterative training scheme for data efficiency. Extensive experimental results in the U.S. market and the China market demonstrate that HRPM achieves significant improvement against many state-of-the-art approaches.

cs.AI↗

arXiv2020

Highly transparent contacts to the 1D hole gas in ultra-scaled Ge/Si core/shell nanowires

Semiconductor-superconductor hybrid systems have outstanding potential for emerging high-performance nanoelectronics and quantum devices. However, critical to their successful application is the fabrication of high-quality and reproducible semiconductor-superconductor interfaces. Here, we realize and measure axial Al-Ge-Al nanowire heterostructures with atomically precise interfaces, enwrapped by an ultrathin epitaxial Si layer further denoted as Al-Ge/Si-Al nanowire heterostructures. The heterostructures were synthesized by a thermally induced exchange reaction of single-crystalline Ge/Si core/shell nanowires and lithographically defined Al contact pads. Applying this heterostructure formation scheme enables self-aligned quasi one-dimensional crystalline Al leads contacting ultrascaled Ge/Si segments with contact transparencies greater than 96%. Integration into back-gated field-effect devices and continuous scaling beyond lithographic limitations allows us to exploit the full potential of the highly transparent contacts to the 1D hole gas at the Ge-Si interface. This leads to the observation of ballistic transport as well as quantum confinement effects up to temperatures of 150 K. Low-temperature measurements reveal proximity-induced superconductivity in the Ge/Si core/shell nanowires. The realization of a Josephson field-effect transistor allows us to study the subgap structure caused by multiple Andreev reflections. Most importantly, the absence of a quantum dot regime indicates a hard superconducting gap originating from the highly transparent contacts to the 1D hole gas, which is potentially interesting for the study of Majorana zero modes. Moreover, underlining the importance of the proposed thermally induced Al-Ge/Si-Al heterostructure formation technique, our system could contribute to the development of key components of quantum computing such as gatemon or transmon qubits

cond-mat.mes-hall↗

arXiv2020

Spin filtering in germanium/silicon core/shell nanowires with pseudo-helical gap

Semiconductors with strong spin-orbit interactions can exhibit a helical gap with spin-momentum locking opened by a magnetic field. Such a gap is highly spin selective as a result of a topologically protected spin-momentum locking, which can be used for spin filtering. We experimentally demonstrate such a spin filtering effect in a quasi-ballistic p-type germanium/silicon core/shell nanowire (NW), which possesses a pseudo-helical gap without the application of magnetic field. Polarized hole spin injection to the NW is achieved using cobalt ferromagnetic contacts with controlled natural surface oxide on the NW as a tunnel barrier. Local and nonlocal spin valve effects are measured as the verification of polarized spin transport in the NW outside the helical gap. We electrically tune the NW into the helical gap by scanning its chemical potential with a gate. A hysteresis loop with three resistance states is observed in the local spin valve geometry, as an evidence of spin filtering in the helical gap.

cond-mat.mes-hall↗

arXiv2020

Correction of Faulty Background Knowledge based on Condition Aware and Revise Transformer for Question Answering

The study of question answering has received increasing attention in recent years. This work focuses on providing an answer that compatible with both user intent and conditioning information corresponding to the question, such as delivery status and stock information in e-commerce. However, these conditions may be wrong or incomplete in real-world applications. Although existing question answering systems have considered the external information, such as categorical attributes and triples in knowledge base, they all assume that the external information is correct and complete. To alleviate the effect of defective condition values, this paper proposes condition aware and revise Transformer (CAR-Transformer). CAR-Transformer (1) revises each condition value based on the whole conversation and original conditions values, and (2) it encodes the revised conditions and utilizes the conditions embedding to select an answer. Experimental results on a real-world customer service dataset demonstrate that the CAR-Transformer can still select an appropriate reply when conditions corresponding to the question exist wrong or missing values, and substantially outperforms baseline models on automatic and human evaluations. The proposed CAR-Transformer can be extended to other NLP tasks which need to consider conditioning information.

cs.CL↗

arXiv2020

A fully discrete energy stable scheme for a phase-field moving contact line model with variable densities and viscosities

In this work, we propose a fully discrete energy stable scheme for the phase-field moving contact line model with variable densities and viscosities. The mathematical model consists of a Cahn-Hilliard equation, a Navier-Stokes equation and the generalized Navier boundary condition for the moving contact line. A scalar auxiliary variable is adopted to transform the governing system into an equivalent form, allowing the double well potential to be treated semi-explicitly. A stabilization term is added to balance the explicit nonlinear term originating from the surface energy at fluid-solid interface. A pressure stabilization method is used to decouple the computation of velocity and pressure. Some subtle implicit-explicit treatments are adopted to deal with convention and stress terms. We establish a rigorous proof of energy stability for the proposed time-marching scheme. Then a finite difference method on staggered grids is used to spatially discretize the constructed time-marching scheme. We further prove that the fully discrete scheme also satisfies the discrete energy dissipation law. Numerical results demonstrate accuracy and energy stability of the proposed scheme. Using our numerical scheme, we analyze the contact line dynamics through a shear flow driven droplet sliding case. Three-dimensional droplet spreading is also investigated on a chemically patterned surface. Our numerical simulation accurately predicts the expected energy evolutions and it successfully reproduces expected phenomena that an oil droplet contracts inwards on a hydrophobic zone and spreads outwards quickly on a hydrophilic zone.

physics.comp-ph↗