arXiv subjects

Felix Yu

Publications and source records attributed to Felix Yu.

At least 37 records · Page 2Linked to original sources

arXiv2023

A Lighter QCD Axion from Anarchy

We introduce the Anarchic Axion, a class of axion models which solves the Strong CP problem within current nEDM constraints with a lighter than usual QCD axion, thus populating new parameter space that ongoing and future experiments target. The Anarchic Axion is driven light by a soft breaking of the Peccei-Quinn symmetry, which also predicts a residual neutron electric dipole moment. We introduce a novel measure to quantify the tuning required for large deviations from the usual QCD axion band. In addition to motivating searches for unusually light axions, this work establishes a new target for axion effective field theory.

hep-ph↗

arXiv2023

Quantum Imprint of the Anharmonic Oscillator

We study the anharmonic double well in quantum mechanics using exact Wentzel-Kramers-Brillouin (WKB) methods in a 't Hooft-like double scaling limit where classical behavior is expected to dominate. We compute the tunneling action in this double scaling limit, and compare it to the transition amplitude from the vacuum to a highly excited state. Our results, exact in the semiclassical limit, show that the two expressions coincide, apart from an irreducible and surprising instanton contribution. Thus, the semiclassical limit of the anharmonic oscillator betrays its quantum origin as a rule, which we dub the "quantum imprint rule," showing that the quantum theory is intrinsically gapped from classical behavior. Besides an example of the failure of reductionism and an example of a resurgent connection between perturbative and nonperturbative physics, this work provides a possible classification of theories according to their quantum imprints.

hep-th↗

arXiv2023

The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers

This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the intermediate output of the multi-layer perceptrons (MLPs) after a ReLU activation function, and by sparse we mean that on average very few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each input to MLP. Moreover, larger Transformers with more layers and wider MLP hidden dimensions are sparser as measured by the percentage of nonzero entries. Through extensive experiments we demonstrate that the emergence of sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks, on both training and evaluation data, for Transformers of various configurations, at layers of all depth levels, as well as for other architectures including MLP-mixers and 2-layer MLPs. We show that sparsity also emerges using training datasets with random labels, or with random inputs, or with infinite amount of data, demonstrating that sparsity is not a result of a specific family of datasets. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers. Moreover, we demonstrate perhaps surprisingly that enforcing an even sparser activation via Top-k thresholding with a small value of k brings a collection of desired but missing properties for Transformers, namely less sensitivity to noisy training data, more robustness to input corruptions, and better calibration for their prediction confidence.

cs.LG↗

arXiv2022

Axion Couplings in Gauged $U(1)'$ Extensions of the Standard Model

We explore the effective theory of an axion in a gauged baryon number symmetry extension of the Standard Model (SM), where the axion is realized from a Dine-Fischler-Srednicki-Zhitnitsky (DFSZ) model construction. Integrating out the anomalons realizes a pattern of effective Wilson coefficients reflecting the factorization between the anomalous Peccei-Quinn and gauged baryon number symmetries. We construct and analyze the chiral transformation invariance of the axion effective theory, accounting for possible flavor-violating axion couplings. We calculate and study the unique phenomenology of the axion and $Z'$ boson, and we present the current collider limits on these particles in the $\{m_{Z'}, g_B\}$, $\{m_a, G_{a γγ} \}$, and $\{m_a, f_a^{-1}\}$ planes.

hep-ph↗

arXiv2022

Large Language Models with Controllable Working Memory

Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP), owing to their excellent understanding and generation abilities. Remarkably, what further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. While many downstream applications provide the model with an informational context to aid its performance on the underlying task, how the model's world knowledge interacts with the factual information presented in the context remains under explored. As a desirable behavior, an LLM should give precedence to the context whenever it contains task-relevant information that conflicts with the model's memorized knowledge. This enables model predictions to be grounded in the context, which can then be used to update or correct specific model predictions without frequent retraining. By contrast, when the context is irrelevant to the task, the model should ignore it and fall back on its internal knowledge. In this paper, we undertake a first joint study of the aforementioned two properties, namely controllability and robustness, in the context of LLMs. We demonstrate that state-of-the-art T5 and PaLM (both pretrained and finetuned) could exhibit poor controllability and robustness, which do not scale with increasing model size. As a solution, we propose a novel method - Knowledge Aware FineTuning (KAFT) - to strengthen both controllability and robustness by incorporating counterfactual and irrelevant contexts to standard supervised datasets. Our comprehensive evaluation showcases the utility of KAFT across model architectures and sizes.

cs.CL↗

arXiv2022

Report of the Topical Group on Physics Beyond the Standard Model at Energy Frontier for Snowmass 2021

This is the Snowmass2021 Energy Frontier (EF) Beyond the Standard Model (BSM) report. It combines the EF topical group reports of EF08 (Model-specific explorations), EF09 (More general explorations), and EF10 (Dark Matter at Colliders). The report includes a general introduction to BSM motivations and the comparative prospects for proposed future experiments for a broad range of potential BSM models and signatures, including compositeness, SUSY, leptoquarks, more general new bosons and fermions, long-lived particles, dark matter, charged-lepton flavor violation, and anomaly detection.

hep-ph↗

arXiv2022

Supersizing axions with small size instantons

We construct a new framework to calculate the enhancement of axion masses and concomitant effects on axion-meson mixing arising from small size instantons (SSIs), which originate in models featuring an extended color gauge symmetry. The framework is based on an explicit evaluation of 't Hooft determinantal operators that partition into instanton amplitudes, affording a more precise determination of the axion-diphoton coupling than previous results. Using an explicit model first presented in Ref. arXiv:1805.06465, we demonstrate that axions solving the strong CP problem can have electroweak scale masses and higher, driven by SSI effects. Such collider axions are prime targets for resonance searches at the Large Hadron Collider and afford a unique anchor for motivating extended color symmetries.

hep-ph↗

arXiv2022

FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning

Federated learning~(FL) has recently attracted increasing attention from academia and industry, with the ultimate goal of achieving collaborative training under privacy and communication constraints. Existing iterative model averaging based FL algorithms require a large number of communication rounds to obtain a well-performed model due to extremely unbalanced and non-i.i.d data partitioning among different clients. Thus, we propose FedDM to build the global training objective from multiple local surrogate functions, which enables the server to gain a more global view of the loss landscape. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data through distribution matching. FedDM reduces communication rounds and improves model quality by transmitting more informative and smaller synthesized data compared with unwieldy model weights. We conduct extensive experiments on three image classification datasets, and results show that our method can outperform other FL counterparts in terms of efficiency and model performance. Moreover, we demonstrate that FedDM can be adapted to preserve differential privacy with Gaussian mechanism and train a better model under the same privacy budget.

cs.LG↗

arXiv2022

Correlated quantization for distributed mean estimation and optimization

We study the problem of distributed mean estimation and optimization under communication constraints. We propose a correlated quantization protocol whose leading term in the error guarantee depends on the mean deviation of data points rather than only their absolute range. The design doesn't need any prior knowledge on the concentration property of the dataset, which is required to get such dependence in previous works. We show that applying the proposed protocol as sub-routine in distributed optimization algorithms leads to better convergence rates. We also prove the optimality of our protocol under mild assumptions. Experimental results show that our proposed algorithm outperforms existing mean estimation protocols on a diverse set of tasks.

cs.LG↗

arXiv2022

InFillmore: Frame-Guided Language Generation with Bidirectional Context

We propose a structured extension to bidirectional-context conditional language generation, or "infilling," inspired by Frame Semantic theory (Fillmore, 1976). Guidance is provided through two approaches: (1) model fine-tuning, conditioning directly on observed symbolic frames, and (2) a novel extension to disjunctive lexically constrained decoding that leverages frame semantic lexical units. Automatic and human evaluations confirm that frame-guided generation allows for explicit manipulation of intended infill semantics, with minimal loss in distinguishability from human-generated text. Our methods flexibly apply to a variety of use scenarios, and we provide a codebase and interactive demo available from https://nlp.jhu.edu/demos/infillmore.

cs.CL↗

arXiv2022

Displaying dark matter constraints from colliders with varying simplified model parameters

The search for dark matter is one of the main science drivers of the particle and astroparticle physics communities. Determining the nature of dark matter will require a broad approach, with a range of experiments pursuing different experimental hypotheses. Within this search program, collider experiments provide insights on dark matter which are complementary to direct/indirect detection experiments and to astrophysical evidence. To compare results from a wide variety of experiments, a common theoretical framework is required. The ATLAS and CMS experiments have adopted a set of simplified models which introduce two new particles, a dark matter particle and a mediator, and whose interaction strengths are set by the couplings of the mediator. So far, the presentation of LHC and future hadron collider results has focused on four benchmark scenarios with specific coupling values within these simplified models. In this work, we describe ways to extend those four benchmark scenarios to arbitrary couplings, and release the corresponding code for use in further studies. This will allow for more straightforward comparison of collider searches to accelerator experiments that are sensitive to smaller couplings, such as those for the US Community Study on the Future of Particle Physics (Snowmass 2021), and will give a more complete picture of the coupling dependence of dark matter collider searches when compared to direct and indirect detection searches. By using semi-analytical methods to rescale collider limits, we drastically reduce the computing resources needed relative to traditional approaches based on the generation of additional simulated signal samples.

hep-ph↗

arXiv2021

HD-cos Networks: Efficient Neural Architectures for Secure Multi-Party Computation

Multi-party computation (MPC) is a branch of cryptography where multiple non-colluding parties execute a well designed protocol to securely compute a function. With the non-colluding party assumption, MPC has a cryptographic guarantee that the parties will not learn sensitive information from the computation process, making it an appealing framework for applications that involve privacy-sensitive user data. In this paper, we study training and inference of neural networks under the MPC setup. This is challenging because the elementary operations of neural networks such as the ReLU activation function and matrix-vector multiplications are very expensive to compute due to the added multi-party communication overhead. To address this, we propose the HD-cos network that uses 1) cosine as activation function, 2) the Hadamard-Diagonal transformation to replace the unstructured linear transformations. We show that both of the approaches enjoy strong theoretical motivations and efficient computation under the MPC setup. We demonstrate on multiple public datasets that HD-cos matches the quality of the more expensive baselines.

cs.CR↗

arXiv2021

Self-supervised Learning for Large-scale Item Recommendations

Large scale recommender models find most relevant items from huge catalogs, and they play a critical role in modern search and recommendation systems. To model the input space with large-vocab categorical features, a typical recommender model learns a joint embedding space through neural networks for both queries and items from user feedback data. However, with millions to billions of items in the corpus, users tend to provide feedback for a very small set of them, causing a power-law distribution. This makes the feedback data for long-tail items extremely sparse. Inspired by the recent success in self-supervised representation learning research in both computer vision and natural language understanding, we propose a multi-task self-supervised learning (SSL) framework for large-scale item recommendations. The framework is designed to tackle the label sparsity problem by learning better latent relationship of item features. Specifically, SSL improves item representation learning as well as serving as additional regularization to improve generalization. Furthermore, we propose a novel data augmentation method that utilizes feature correlations within the proposed framework. We evaluate our framework using two real-world datasets with 500M and 1B training examples respectively. Our results demonstrate the effectiveness of SSL regularization and show its superior performance over the state-of-the-art regularization techniques. We also have already launched the proposed techniques to a web-scale commercial app-to-app recommendation system, with significant improvements top-tier business metrics demonstrated in A/B experiments on live traffic. Our online results also verify our hypothesis that our framework indeed improves model performance even more on slices that lack supervision.

cs.LG↗

arXiv2021

Learning discrete distributions: user vs item-level privacy

Much of the literature on differential privacy focuses on item-level privacy, where loosely speaking, the goal is to provide privacy per item or training example. However, recently many practical applications such as federated learning require preserving privacy for all items of a single user, which is much harder to achieve. Therefore understanding the theoretical limit of user-level privacy becomes crucial. We study the fundamental problem of learning discrete distributions over $k$ symbols with user-level differential privacy. If each user has $m$ samples, we show that straightforward applications of Laplace or Gaussian mechanisms require the number of users to be $\mathcal{O}(k/(mα^2) + k/εα)$ to achieve an $\ell_1$ distance of $α$ between the true and estimated distributions, with the privacy-induced penalty $k/εα$ independent of the number of samples per user $m$. Moreover, we show that any mechanism that only operates on the final aggregate counts should require a user complexity of the same order. We then propose a mechanism such that the number of users scales as $\tilde{\mathcal{O}}(k/(mα^2) + k/\sqrt{m}εα)$ and hence the privacy penalty is $\tildeΘ(\sqrt{m})$ times smaller compared to the standard mechanisms in certain settings of interest. We further show that the proposed mechanism is nearly-optimal under certain regimes. We also propose general techniques for obtaining lower bounds on restricted differentially private estimators and a lower bound on the total variation between binomial distributions, both of which might be of independent interest.

cs.LG↗

arXiv2020

Modifying Memories in Transformer Models

Large Transformer models have achieved impressive performance in many natural language tasks. In particular, Transformer based language models have been shown to have great capabilities in encoding factual knowledge in their vast amount of parameters. While the tasks of improving the memorization and generalization of Transformers have been widely studied, it is not well known how to make transformers forget specific old facts and memorize new ones. In this paper, we propose a new task of \emph{explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts}. This task is useful in many scenarios, such as updating stale knowledge, protecting privacy, and eliminating unintended biases stored in the models. We benchmarked several approaches that provide natural baseline performances on this task. This leads to the discovery of key components of a Transformer model that are especially effective for knowledge modifications. The work also provides insights into the role that different training phases (such as pretraining and fine-tuning) play towards memorization and knowledge modification.

cs.CL↗

arXiv2020

Semantic Label Smoothing for Sequence to Sequence Problems

Label smoothing has been shown to be an effective regularization strategy in classification, that prevents overfitting and helps in label de-noising. However, extending such methods directly to seq2seq settings, such as Machine Translation, is challenging: the large target output space of such problems makes it intractable to apply label smoothing over all possible outputs. Most existing approaches for seq2seq settings either do token level smoothing, or smooth over sequences generated by randomly substituting tokens in the target sequence. Unlike these works, in this paper, we propose a technique that smooths over \emph{well formed} relevant sequences that not only have sufficient n-gram overlap with the target sequence, but are also \emph{semantically similar}. Our method shows a consistent and significant improvement over the state-of-the-art techniques on different datasets.

cs.CL↗

arXiv2020

Probing new $U(1)$ gauge symmetries via exotic $Z \to Z' γ$ decays

New $U(1)$ gauge theories involving Standard Model (SM) fermions typically require additional electroweak fermions for anomaly cancellation. We study the non-decoupling properties of these new fermions, called anomalons, in the $Z-Z'-γ$ vertex function, reviewing the connection between the full model and the effective Wess-Zumino operator. We calculate the exotic $Z \to Z' γ$ decay width in $U(1)_{B-L}$ and $U(1)_B$ models, where $B$ and $L$ denote the SM baryon and lepton number symmetries. For $U(1)_{B-L}$ gauge symmetry, each generation of SM fermions is anomaly free and the exotic $Z \to Z_{BL}' γ$ decay width is entirely induced by intragenerational mass splittings. In contrast, for $U(1)_B$ gauge symmetry, the existence of two distinct sources of chiral symmetry breaking enables a heavy, anomaly-free set of fermions to have an irreducible contribution to the $Z \to Z_B' γ$ decay width. We show that the current LEP limits on the exotic $Z \to Z'_B γ$ decay are weaker than previously estimated, and low-mass $Z'_B$ dijet resonance searches are currently more constraining. We present a summary of the current collider bounds on $U(1)_B$ and a projection for a TeraZ factory on the $Z \to Z_B' γ$ exotic decay, and emphasize how the $Z \to Z' γ$ decay is emblematic of new anomalous $U(1)$ gauge symmetries.

hep-ph↗

arXiv2020

Doubly-stochastic mining for heterogeneous retrieval

Modern retrieval problems are characterised by training sets with potentially billions of labels, and heterogeneous data distributions across subpopulations (e.g., users of a retrieval system may be from different countries), each of which poses a challenge. The first challenge concerns scalability: with a large number of labels, standard losses are difficult to optimise even on a single example. The second challenge concerns uniformity: one ideally wants good performance on each subpopulation. While several solutions have been proposed to address the first challenge, the second challenge has received relatively less attention. In this paper, we propose doubly-stochastic mining (S2M ), a stochastic optimization technique that addresses both challenges. In each iteration of S2M, we compute a per-example loss based on a subset of hardest labels, and then compute the minibatch loss based on the hardest examples. We show theoretically and empirically that by focusing on the hardest examples, S2M ensures that all data subpopulations are modelled well.

cs.LG↗