arXiv subjects

Fei Sun

Publications and source records attributed to Fei Sun.

At least 91 records · Page 5Linked to original sources

arXiv2024

Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts

Weight-sharing supernets are crucial for performance estimation in cutting-edge neural architecture search (NAS) frameworks. Despite their ability to generate diverse subnetworks without retraining, the quality of these subnetworks is not guaranteed due to weight sharing. In NLP tasks like machine translation and pre-trained language modeling, there is a significant performance gap between supernet and training from scratch for the same model architecture, necessitating retraining post optimal architecture identification. This study introduces a solution called mixture-of-supernets, a generalized supernet formulation leveraging mixture-of-experts (MoE) to enhance supernet model expressiveness with minimal training overhead. Unlike conventional supernets, this method employs an architecture-based routing mechanism, enabling indirect sharing of model weights among subnetworks. This customization of weights for specific architectures, learned through gradient descent, minimizes retraining time, significantly enhancing training efficiency in NLP. The proposed method attains state-of-the-art (SoTA) performance in NAS for fast machine translation models, exhibiting a superior latency-BLEU tradeoff compared to HAT, the SoTA NAS framework for machine translation. Furthermore, it excels in NAS for building memory-efficient task-agnostic BERT models, surpassing NAS-BERT and AutoDistil across various model sizes. The code can be found at: https://github.com/UBC-NLP/MoS.

cs.CL↗

arXiv2024

Broadband planar electromagnetic hyper-lens with uniform magnification in air

A planar hyper-lens, capable of creating sub-wavelength imaging for broadband electromagnetic wave, is designed based on electromagnetic null medium. Subsequently, a scheme for the implementation of the proposed hyper-lens is given by using well-designed flexural metal plates, which function as the reduced electromagnetic null medium for TM-polarized microwaves. Both simulated and measured results verify that the hyper-lens designed with flexural metal plates can achieve super-resolution imaging for microwave at operating wavelength (λ0=3cm) with a resolution of 0.25λ0 and a uniform magnification of about 5. Moreover, the designed hyper-lens ensures that both the object and image surfaces are planes, and simultaneously provides a uniform magnification for objects in different positions. Additionally, the proposed hyper-lens offers a broadband super-resolution imaging capabilities, achieving good super-resolution imaging effects for microwave frequencies ranging from 8.5 to 11 GHz. The proposed hyper-lens may find applications in high precision imaging, detection, and sensing.

physics.app-ph↗

arXiv2024

Photoproduction of lepton pair in ultra-relativistic heavy-ion collisions

Dilepton production provides a unique probe of the strong electromagnetic field produced in heavy-ion collisions. To map out the behavior of its transverse momentum broadening, we present a theoretical model based on the equivalent photon approximation, and then we update it to make direct comparisons with the recent experimental measurements. We find that the model calculations can describe well, not only the average transverse momentum squared of $e^{+}e^{-}$ pairs in Au--Au collisions at $\sqrt{s_{\rm NN}}=200$ GeV, but also the acoplanarity of $μ^{+}μ^{-}$ pairs in Pb--Pb collisions at$\sqrt{s_{\rm NN}}=5.02$ TeV. Furthermore, the model predictions are also able to reproduce the measured dependencies of the pair mass and the transverse momentum squared.

nucl-th↗

arXiv2024

Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts?

While auxiliary information has become a key to enhancing Large Language Models (LLMs), relatively little is known about how LLMs merge these contexts, specifically contexts generated by LLMs and those retrieved from external sources. To investigate this, we formulate a systematic framework to identify whether LLMs' responses are attributed to either generated or retrieved contexts. To easily trace the origin of the response, we construct datasets with conflicting contexts, i.e., each question is paired with both generated and retrieved contexts, yet only one of them contains the correct answer. Our experiments reveal a significant bias in several LLMs (GPT-4/3.5 and Llama2) to favor generated contexts, even when they provide incorrect information. We further identify two key factors contributing to this bias: i) contexts generated by LLMs typically show greater similarity to the questions, increasing their likelihood of being selected; ii) the segmentation process used in retrieved contexts disrupts their completeness, thereby hindering their full utilization in LLMs. Our analysis enhances the understanding of how LLMs merge diverse contexts, offers valuable insights for advancing current LLM augmentation methods, and highlights the risk of generated misinformation for retrieval-augmented LLMs.

cs.CL↗

arXiv2024

The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse

Although model editing has shown promise in revising knowledge in Large Language Models (LLMs), its impact on the inherent capabilities of LLMs is often overlooked. In this work, we reveal a critical phenomenon: even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. However, benchmarking LLMs after each edit, while necessary to prevent such collapses, is impractically time-consuming and resource-intensive. To mitigate this, we propose using perplexity as a surrogate metric, validated by extensive experiments demonstrating changes in an edited model's perplexity are strongly correlated with its downstream task performances. We further conduct an in-depth study on sequential editing, a practical setting for real-world scenarios, across various editing methods and LLMs, focusing on hard cases from our previous single edit studies. The results indicate that nearly all examined editing methods result in model collapse after only few edits. To facilitate further research, we have utilized GPT-3.5 to develop a new dataset, HardEdit, based on those hard cases. This dataset aims to establish the foundation for pioneering research in reliable model editing and the mechanisms underlying editing-induced model collapse. We hope this work can draw the community's attention to the potential risks inherent in model editing practices.

cs.AI↗

arXiv2024

Impurity-level induced broadband photoelectric response in wide-band semiconductor SrSnO3

Broadband spectrum detectors exhibit great promise in fields such as multispectral imaging and optical communications. Despite significant progress, challenges like materials instability, complex manufacturing process and high costs still hinder further application. Here we present a method that achieves broadband spectral detect by impurity-level in SrSnO3. We report over 200 mA/W photo-responsivity at 275 nm (ultraviolet C solar-bind) and 367 nm (ultraviolet A) and ~ 1 mA/W photo-responsivity at 532 nm and 700 nm (visible) with a voltage bias of 5V. Further transport and photoluminescence results indicate that the broadband response comes from the impurity levels and mutual interactions. Additionally, the photodetector demonstrates excellent robustness and stability under repeated tests and prolonged exposure in air. These findings show the potential of SSO photodetectors and propose a method to achieve broadband spectrum detection, creating new possibility for the development of single-phase, low-cost, simple structure and high-efficiency photodetectors.

physics.app-ph↗

arXiv2024

Generative AI Beyond LLMs: System Implications of Multi-Modal Generation

As the development of large-scale Generative AI models evolve beyond text (1D) generation to include image (2D) and video (3D) generation, processing spatial and temporal information presents unique challenges to quality, performance, and efficiency. We present the first work towards understanding this new system design space for multi-modal text-to-image (TTI) and text-to-video (TTV) generation models. Current model architecture designs are bifurcated into 2 categories: Diffusion- and Transformer-based models. Our systematic performance characterization on a suite of eight representative TTI/TTV models shows that after state-of-the-art optimization techniques such as Flash Attention are applied, Convolution accounts for up to 44% of execution time for Diffusion-based TTI models, while Linear layers consume up to 49% of execution time for Transformer-based models. We additionally observe that Diffusion-based TTI models resemble the Prefill stage of LLM inference, and benefit from 1.1-2.5x greater speedup from Flash Attention than Transformer-based TTI models that resemble the Decode phase. Since optimizations designed for LLMs do not map directly onto TTI/TTV models, we must conduct a thorough characterization of these workloads to gain insights for new optimization opportunities. In doing so, we define sequence length in the context of TTI/TTV models and observe sequence length can vary up to 4x in Diffusion model inference. We additionally observe temporal aspects of TTV workloads pose unique system bottlenecks, with Temporal Attention accounting for over 60% of total Attention time. Overall, our in-depth system performance characterization is a critical first step towards designing efficient and deployable systems for emerging TTI/TTV workloads.

cs.DC↗

arXiv2024

Is Flash Attention Stable?

Training large-scale machine learning models poses distinct system challenges, given both the size and complexity of today's workloads. Recently, many organizations training state-of-the-art Generative AI models have reported cases of instability during training, often taking the form of loss spikes. Numeric deviation has emerged as a potential cause of this training instability, although quantifying this is especially challenging given the costly nature of training runs. In this work, we develop a principled approach to understanding the effects of numeric deviation, and construct proxies to put observations into context when downstream effects are difficult to quantify. As a case study, we apply this framework to analyze the widely-adopted Flash Attention optimization. We find that Flash Attention sees roughly an order of magnitude more numeric deviation as compared to Baseline Attention at BF16 when measured during an isolated forward pass. We then use a data-driven analysis based on the Wasserstein Distance to provide upper bounds on how this numeric deviation impacts model weights during training, finding that the numerical deviation present in Flash Attention is 2-5 times less significant than low-precision training.

cs.LG↗

arXiv2024

Experimental demonstration of a thermal-EM concentrator for enhancing EM signals and converging heat fluxes simultaneously

Simultaneously concentrating EM waves and heat fluxes to the same target region within an on-chip system carries substantial academic research importance and practical application value. Nevertheless, existing researches are primarily aimed at the design and experimentation of concentrators for individual EM waves or temperature fields. In this work, a thermal-EM concentrator, capable of simultaneously concentrating EM waves and heat fluxes, is designed using transformation optics/thermodynamics and fabricated with engineered EM-thermal metamaterials. The concentrating effects of the proposed thermal-EM concentrator on the thermal fluxes and EM waves are verified through numerical simulations and experimental measurements, respectively, which are in good agreement with each other. Both numerically simulated and experimentally measured results demonstrate the concentrating capability of the proposed thermal-EM concentrator, which can concentrate broadband TM-polarized EM waves ranging from 8-12 GHz and heat/cold flows to the same target region within an on-chip operating environment. The thermal-EM concentrator exhibits a thermal focusing efficiency close to 100% and more than three times enhancement of the magnetic field at the designed center frequency of 10 GHz. The proposed thermal-EM concentrator can be utilized for efficient cooling for the specified component and simultaneously enhancing the EM antenna's radiation/reception efficiency within an on-chip system.

physics.gen-ph↗

arXiv2024

Chiral phase transition and spin alignment of vector meson in the Polarized-Polyakov-loop Nambu-Jona-Lasinio model under rotation

By using the extrapolation method, a polarized Polykov-loop potential at finite real angular velocity is constructed from the lattice results at finite imaginary angular velocity. The chiral and deconfinement phase transitions under rotation have been simultaneously investigated in the Polarized-Polyakov-loop Nambu-Jona-Lasinio (PPNJL) model. It is observed that both critical temperatures of deconfinement and chiral phase transition increase with the angular velocity, which is in consistent with lattice results. The spin alignment of vector meson has the negative deviation of $ρ_{00} -1/3$ under rotation, and the deviation in the PPNJL model is much more significant than that in the NJL model and the quark coalescence model, which revealing the important role of rotating gluons on the quark polarization.

hep-ph↗

arXiv2024

Unlink to Unlearn: Simplifying Edge Unlearning in GNNs

As concerns over data privacy intensify, unlearning in Graph Neural Networks (GNNs) has emerged as a prominent research frontier in academia. This concept is pivotal in enforcing the \textit{right to be forgotten}, which entails the selective removal of specific data from trained GNNs upon user request. Our research focuses on edge unlearning, a process of particular relevance to real-world applications. Current state-of-the-art approaches like GNNDelete can eliminate the influence of specific edges yet suffer from \textit{over-forgetting}, which means the unlearning process inadvertently removes excessive information beyond needed, leading to a significant performance decline for remaining edges. Our analysis identifies the loss functions of GNNDelete as the primary source of over-forgetting and also suggests that loss functions may be redundant for effective edge unlearning. Building on these insights, we simplify GNNDelete to develop \textbf{Unlink to Unlearn} (UtU), a novel method that facilitates unlearning exclusively through unlinking the forget edges from graph structure. Our extensive experiments demonstrate that UtU delivers privacy protection on par with that of a retrained model while preserving high accuracy in downstream tasks, by upholding over 97.3\% of the retrained model's privacy protection capabilities and 99.8\% of its link prediction accuracy. Meanwhile, UtU requires only constant computational demands, underscoring its advantage as a highly lightweight and practical edge unlearning solution.

cs.LG↗

arXiv2024

TEA: Test-time Energy Adaptation

Test-time adaptation (TTA) aims to improve model generalizability when test data diverges from training distribution, offering the distinct advantage of not requiring access to training data and processes, especially valuable in the context of large pre-trained models. However, current TTA methods fail to address the fundamental issue: covariate shift, i.e., the decreased generalizability can be attributed to the model's reliance on the marginal distribution of the training data, which may impair model calibration and introduce confirmation bias. To address this, we propose a novel energy-based perspective, enhancing the model's perception of target data distributions without requiring access to training data or processes. Building on this perspective, we introduce $\textbf{T}$est-time $\textbf{E}$nergy $\textbf{A}$daptation ($\textbf{TEA}$), which transforms the trained classifier into an energy-based model and aligns the model's distribution with the test data's, enhancing its ability to perceive test distributions and thus improving overall generalizability. Extensive experiments across multiple tasks, benchmarks and architectures demonstrate TEA's superior generalization performance against state-of-the-art methods. Further in-depth analyses reveal that TEA can equip the model with a comprehensive perception of test distribution, ultimately paving the way toward improved generalization and calibration.

cs.LG↗

arXiv2024

Tunable uniform field enhancement in a subwavelength air pillar by photonic doping in epsilon-near-zero medium

In this study, a novel electric field compressor is proposed by doping a metal-air-metal pillar in epsilon-near-zero medium within a metallic waveguide, which effectively enhances the background electric fields in a sub-wavelength air pillar with high uniformity. The field enhancement factor can be analytically determined through theoretical derivations from Maxwell's equations, reaching its maximum value by appropriately selecting the size of the air pillar. Furthermore, the proposed compressor can achieve a tunable electric field enhancement effect within the deep sub-wavelength air pillar by adjusting the height of the air pillar through movement of two metal pillars inserted into the waveguide. Both theoretical analysis and numerical simulations are employed to validate the performance of the electric field compressor, which exhibits a wide range of tunable field enhancement effect (i.e., continuously adjustable enhancement factor between 20 and 800) with uniformity below 1.5 within the deep sub-wavelength air pillar (wherein the air volume is smaller than la0/10 * la0/10 * la0/16). Finally, a practical model is proposed for the realization of the electric field compressor in the microwave range, where specially placed metal pillars and wires are incorporated into a rectangular metallic waveguide structure. The field enhancement and tunable effects of this practical model have been verified through simulations.

physics.app-ph↗

arXiv2024

Toward Practical Entity Alignment Method Design: Insights from New Highly Heterogeneous Knowledge Graph Datasets

The flourishing of knowledge graph applications has driven the need for entity alignment (EA) across KGs. However, the heterogeneity of practical KGs, characterized by differing scales, structures, and limited overlapping entities, greatly surpasses that of existing EA datasets. This discrepancy highlights an oversimplified heterogeneity in current EA datasets, which obstructs a full understanding of the advancements achieved by recent EA methods. In this paper, we study the performance of EA methods in practical settings, specifically focusing on the alignment of highly heterogeneous KGs (HHKGs). Firstly, we address the oversimplified heterogeneity settings of current datasets and propose two new HHKG datasets that closely mimic practical EA scenarios. Then, based on these datasets, we conduct extensive experiments to evaluate previous representative EA methods. Our findings reveal that, in aligning HHKGs, valuable structure information can hardly be exploited through message-passing and aggregation mechanisms. This phenomenon leads to inferior performance of existing EA methods, especially those based on GNNs. These findings shed light on the potential problems associated with the conventional application of GNN-based methods as a panacea for all EA datasets. Consequently, in light of these observations and to elucidate what EA methodology is genuinely beneficial in practical scenarios, we undertake an in-depth analysis by implementing a simple but effective approach: Simple-HHEA. This method adaptly integrates entity name, structure, and temporal information to navigate the challenges posed by HHKGs. Our experiment results conclude that the key to the future EA model design in practice lies in their adaptability and efficiency to varying information quality conditions, as well as their capability to capture patterns across HHKGs.

cs.LG↗

arXiv2024

Unraveling collisional energy loss of a heavy quark in quark-gluon plasma

At leading order in QCD coupling constant, we compute the energy loss per traveling distance of a heavy quark $dE/dz$ from elastic scattering off thermal quarks and gluons at a temperature $T$, including the thermal perturbative description of soft scatterings ($-t<-t^{\ast}$) and a perturbative QCD-based calculation for hard collisions ($-t>-t^{\ast}$). Within this soft-hard factorization model, we find that the full results of $dE/dz$ behaves a mild sensitivity to the intermediate cutoff $t^{\ast}$, supporting the validity of the soft-hard approach within the temperature region of interest. We re-derive the analytic formula for $dE/dz$ in the high-energy approximation, $E_{1}\gg m^{2}_{1}/T$, where $E_{1}$ is the injected heavy quark energy and $m_{1}$ is its mass. It is realized that the soft logarithmic contribution, $dE/dz\propto ln(-t^{\ast}/m^{2}_{D})$, arises from the $t$-channel scattering off thermal partons, while the hard logarithmic term, $dE/dz\propto ln[E_{1}T/(-t^{\ast})]$, stems from the $t$-channel scattering off thermal partons, and the one $dE/dz\propto ln(E_{1}T/m^{2}_{1})$ comes from the $s$- and $u$-channel scattering off gluons. The sum of these contributions cancels the $t^{\ast}$-dependence as observed in the full result. The mass hierarchy is observed $dE/dz(charm)>dE/dz(bottom)$. Our full results are crucial for a better description of heavy quark transport in QCD medium, in particular at low and moderate energy. We also calculate the energy loss by imposing the Einstein's relationship. The related results appear to be systematically larger than that without imposing the Einstein's relationship.

hep-ph↗

arXiv2023

PDE+: Enhancing Generalization via PDE with Adaptive Distributional Diffusion

The generalization of neural networks is a central challenge in machine learning, especially concerning the performance under distributions that differ from training ones. Current methods, mainly based on the data-driven paradigm such as data augmentation, adversarial training, and noise injection, may encounter limited generalization due to model non-smoothness. In this paper, we propose to investigate generalization from a Partial Differential Equation (PDE) perspective, aiming to enhance it directly through the underlying function of neural networks, rather than focusing on adjusting input data. Specifically, we first establish the connection between neural network generalization and the smoothness of the solution to a specific PDE, namely "transport equation". Building upon this, we propose a general framework that introduces adaptive distributional diffusion into transport equation to enhance the smoothness of its solution, thereby improving generalization. In the context of neural networks, we put this theoretical framework into practice as $\textbf{PDE+}$ ($\textbf{PDE}$ with $\textbf{A}$daptive $\textbf{D}$istributional $\textbf{D}$iffusion) which diffuses each sample into a distribution covering semantically similar inputs. This enables better coverage of potentially unobserved distributions in training, thus improving generalization beyond merely data-driven methods. The effectiveness of PDE+ is validated through extensive experimental settings, demonstrating its superior performance compared to SOTA methods.

cs.LG↗

arXiv2023

On-chip omnidirectional electromagnetic-thermal cloak

Simultaneously guiding electromagnetic waves and heat flow at any incidence angle to smoothly bypass some electromagnetic/thermal sensitive elements is a key factor to ensure efficient communication and thermal protection for an on-chip system. In this study, an omnidirectional on-chip electromagnetic-thermal cloak is proposed. Firstly, a holey metallic plate with periodic array of subwavelength apertures is designed by optical surface transformation to realize an omnidirectional electromagnetic cloaking module for on-chip electromagnetic signal. Secondly, a two-layer ring-shaped engineered thermal structure is designed by solving Laplace equation to realize an omnidirectional thermal cloaking module for in-chip heat flow. Finally, these two cloaking modules are combined elaborately to achieve cloaking effect for both the electromagnetic waves and thermal fields simultaneously from any detecting direction, thus protecting the build-in electromagnetic/thermal sensitive elements without disturbing the external electromagnetic/thermal signal. The proposed electromagnetic-thermal cloak may have potential advantage in dealing with omnidirectional electromagnetic compatibility/shielding and multi-directional thermal management/dissipation of an on-chip system.

physics.app-ph↗

arXiv2023

EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything

Segment Anything Model (SAM) has emerged as a powerful tool for numerous vision applications. A key component that drives the impressive performance for zero-shot transfer and high versatility is a super large Transformer model trained on the extensive high-quality SA-1B dataset. While beneficial, the huge computation cost of SAM model has limited its applications to wider real-world applications. To address this limitation, we propose EfficientSAMs, light-weight SAM models that exhibits decent performance with largely reduced complexity. Our idea is based on leveraging masked image pretraining, SAMI, which learns to reconstruct features from SAM image encoder for effective visual representation learning. Further, we take SAMI-pretrained light-weight image encoders and mask decoder to build EfficientSAMs, and finetune the models on SA-1B for segment anything task. We perform evaluations on multiple vision tasks including image classification, object detection, instance segmentation, and semantic object detection, and find that our proposed pretraining method, SAMI, consistently outperforms other masked image pretraining methods. On segment anything task such as zero-shot instance segmentation, our EfficientSAMs with SAMI-pretrained lightweight image encoders perform favorably with a significant gain (e.g., ~4 AP on COCO/LVIS) over other fast SAM models.

cs.CV↗