arXiv subjects

Michael Wick

Publications and source records attributed to Michael Wick.

12 recordsLinked to original sources

arXiv2024

Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT

In the current landscape of language model research, larger models, larger datasets and more compute seems to be the only way to advance towards intelligence. While there have been extensive studies of scaling laws and models' scaling behaviors, the effect of scale on a model's social biases and stereotyping tendencies has received less attention. In this study, we explore the influence of model scale and pre-training data on its learnt social biases. We focus on BERT -- an extremely popular language model -- and investigate biases as they show up during language modeling (upstream), as well as during classification applications after fine-tuning (downstream). Our experiments on four architecture sizes of BERT demonstrate that pre-training data substantially influences how upstream biases evolve with model scale. With increasing scale, models pre-trained on large internet scrapes like Common Crawl exhibit higher toxicity, whereas models pre-trained on moderated data sources like Wikipedia show greater gender stereotypes. However, downstream biases generally decrease with increasing model scale, irrespective of the pre-training data. Our results highlight the qualitative role of pre-training data in the biased behavior of language models, an often overlooked aspect in the study of scale. Through a detailed case study of BERT, we shed light on the complex interplay of data and model scale, and investigate how it translates to concrete biases.

cs.CL↗

arXiv2021

Conjugate Energy-Based Models

In this paper, we propose conjugate energy-based models (CEBMs), a new class of energy-based models that define a joint density over data and latent variables. The joint density of a CEBM decomposes into an intractable distribution over data and a tractable posterior over latent variables. CEBMs have similar use cases as variational autoencoders, in the sense that they learn an unsupervised mapping from data to latent variables. However, these models omit a generator network, which allows them to learn more flexible notions of similarity between data points. Our experiments demonstrate that conjugate EBMs achieve competitive results in terms of image modelling, predictive power of latent space, and out-of-domain detection on a variety of datasets.

cs.LG↗

arXiv2019

Gradient-based Inference for Networks with Output Constraints

Practitioners apply neural networks to increasingly complex problems in natural language processing, such as syntactic parsing and semantic role labeling that have rich output structures. Many such structured-prediction problems require deterministic constraints on the output values; for example, in sequence-to-sequence syntactic parsing, we require that the sequential outputs encode valid trees. While hidden units might capture such properties, the network is not always able to learn such constraints from the training data alone, and practitioners must then resort to post-processing. In this paper, we present an inference method for neural networks that enforces deterministic constraints on outputs without performing rule-based post-processing or expensive discrete search. Instead, in the spirit of gradient-based training, we enforce constraints with gradient-based inference (GBI): for each input at test-time, we nudge continuous model weights until the network's unconstrained inference procedure generates an output that satisfies the constraints. We study the efficacy of GBI on three tasks with hard constraints: semantic role labeling, syntactic parsing, and sequence transduction. In each case, the algorithm not only satisfies constraints but improves accuracy, even when the underlying network is state-of-the-art.

cs.CL↗

arXiv2018

Sketching for Latent Dirichlet-Categorical Models

Recent work has explored transforming data sets into smaller, approximate summaries in order to scale Bayesian inference. We examine a related problem in which the parameters of a Bayesian model are very large and expensive to store in memory, and propose more compact representations of parameter values that can be used during inference. We focus on a class of graphical models that we refer to as latent Dirichlet-Categorical models, and show how a combination of two sketching algorithms known as count-min sketch and approximate counters provide an efficient representation for them. We show that this sketch combination -- which, despite having been used before in NLP applications, has not been previously analyzed -- enjoys desirable properties. We prove that for this class of models, when the sketches are used during Markov Chain Monte Carlo inference, the equilibrium of sketched MCMC converges to that of the exact chain as sketch parameters are tuned to reduce the error rate.

cs.LG↗

arXiv2011

Theoretical and Phenomenological Constraints on Form Factors for Radiative and Semi-Leptonic B-Meson Decays

We study transition form factors for radiative and rare semi-leptonic B-meson decays into light pseudoscalar or vector mesons, combining theoretical constraints and phenomenological information from Lattice QCD, light-cone sum rules, and dispersive bounds. We pay particular attention to form factor parameterisations which are based on the so-called series expansion, and study the related systematic uncertainties on a quantitative level. In this context, we also provide the NLO corrections to the correlation function between two flavour-changing tensor currents, which enters the unitarity constraints for the coefficients in the series expansion.

hep-ph↗

arXiv2011

Symmetries and Asymmetries of B -> K* mu+ mu- Decays in the Standard Model and Beyond

The rare decay B -> K* (-> K pi) mu+ mu- is regarded as one of the crucial channels for B physics as the polarization of the K* allows a precise angular reconstruction resulting in many observables that offer new important tests of the Standard Model and its extensions. These angular observables can be expressed in terms of CP-conserving and CP-violating quantities which we study in terms of the full form factors calculated from QCD sum rules on the light-cone, including QCD factorization corrections. We investigate all observables in the context of the Standard Model and various New Physics models, in particular the Littlest Higgs model with T-parity and various MSSM scenarios, identifying those observables with small to moderate dependence on hadronic quantities and large impact of New Physics. One important result of our studies is that new CP-violating phases will produce clean signals in CP-violating asymmetries. We also identify a number of correlations between various observables which will allow a clear distinction between different New Physics scenarios.

hep-ph↗

arXiv2010

Distantly Labeling Data for Large Scale Cross-Document Coreference

Cross-document coreference, the problem of resolving entity mentions across multi-document collections, is crucial to automated knowledge base construction and data mining tasks. However, the scarcity of large labeled data sets has hindered supervised machine learning research for this task. In this paper we develop and demonstrate an approach based on ``distantly-labeling'' a data set from which we can train a discriminative cross-document coreference model. In particular we build a dataset of more than a million people mentions extracted from 3.5 years of New York Times articles, leverage Wikipedia for distant labeling with a generative model (and measure the reliability of such labeling); then we train and evaluate a conditional random field coreference model that has factors on cross-document entities as well as mention-pairs. This coreference model obtains high accuracy in resolving mentions and entities that are not present in the training data, indicating applicability to non-Wikipedia data. Given the large amount of data, our work is also an exercise demonstrating the scalability of our approach.

cs.AI↗

arXiv2010

Scalable Probabilistic Databases with Factor Graphs and MCMC

Probabilistic databases play a crucial role in the management and understanding of uncertain data. However, incorporating probabilities into the semantics of incomplete databases has posed many challenges, forcing systems to sacrifice modeling power, scalability, or restrict the class of relational algebra formula under which they are closed. We propose an alternative approach where the underlying relational database always represents a single world, and an external factor graph encodes a distribution over possible worlds; Markov chain Monte Carlo (MCMC) inference is then used to recover this uncertainty to a desired level of fidelity. Our approach allows the efficient evaluation of arbitrary queries over probabilistic databases with arbitrary dependencies expressed by graphical models with structure that changes during inference. MCMC sampling provides efficiency by hypothesizing {\em modifications} to possible worlds rather than generating entire worlds from scratch. Queries are then run over the portions of the world that change, avoiding the onerous cost of running full queries over each sampled world. A significant innovation of this work is the connection between MCMC sampling and materialized view maintenance techniques: we find empirically that using view maintenance techniques is several orders of magnitude faster than naively querying each sampled world. We also demonstrate our system's ability to answer relational queries with aggregation, and demonstrate additional scalability through the use of parallelization.

cs.DB↗

arXiv2009

Probing New Physics with b to s l l and b to s nu nu transitions

The rare decay B to K* (to K pi) mu+ mu- is regarded as one of the crucial channels for B physics since its angular distribution gives access to many observables that offer new important tests of the Standard Model and its extensions. We point out a number of correlations among various observables which will allow a clear distinction between different New Physics (NP) scenarios. Furthermore, we discuss the decay B to K* nu anti-nu which allows for a transparent study of Z penguin effects in NP frameworks in the absence of dipole operator contributions and Higgs penguin contributions. We study all possible observables in B to K* nu anti-nu and the related b to s transitions B to K nu anti-nu and B to X_s nu anti-nu in the context of the SM and various NP models.

hep-ph↗

arXiv2009

New strategies for New Physics search in B -> K* nu anti-nu, B -> K nu anti-nu and B -> X(s) nu anti-nu decays

The rare decay B -> K* nu anti-nu allows a transparent study of Z penguin and other electroweak penguin effects in New Physics (NP) scenarios in the absence of dipole operator contributions and Higgs (scalar) penguin contributions that are often more important than Z contributions in B -> K* l+l- and B(s) -> l+l- decays. We present a new analysis of B -> K* nu anti-nu with improved form factors and of the decays B -> K nu anti-nu and B -> X(s) nu anti-nu in the SM and in a number of NP scenarios like the general MSSM, general scenarios with modified Z/Z' penguins and in a singlet scalar extension of the SM. We also summarize the results in the Littlest Higgs model with T-parity and a Randall-Sundrum (RS) model with custodial protection of left-handed Z-di-dj couplings. Our SM prediction BR(B -> K* nu anti-nu)=(6.8^+1.0_-1.1) x 10^-6 turns out to be significantly lower than the ones present in the literature. Our improved calculation BR(B -> X(s) nu anti-nu)=(2.7+-0.2) x 10^-5 in the SM avoids the normalization to the BR(B -> X(c) e anti-nu(e)) and, with less than 10% total uncertainty, is the most accurate to date. The results for the SM and NP scenarios can be transparently summarized in a (epsilon,eta) plane with a non-vanishing eta signalling the presence of new right-handed down-quark flavour violating couplings which can be ideally probed by the decays in question. Measuring the three branching ratios and one additional polarization observable in B -> K* nu anti-nu allows to overconstrain the resulting point in the (epsilon,eta) plane with (epsilon,eta)=(1,0) corresponding to the SM. The correlations of these three channels with the rare decays K+ -> pi+ nu anti-nu, KL -> pi0 nu anti-nu, B -> X(s) l+ l- and B(s) -> mu+ mu- offer powerful tests of New Physics with new right-handed couplings and non-MFV interactions.

hep-ph↗

arXiv2008

A Reconsideration of the b -> s gamma Decay in the Minimal Flavor Violating MSSM

We present a MSSM study of the b -> s gamma decay in a Minimal Flavor Violating (MFV) framework, where the form of the soft SUSY breaking terms is determined by the Standard Model Yukawa couplings. In particular, we address the role of gluino contributions, which are set to zero in most studies of the MFV MSSM. Gluino contributions can play an important role in the MFV MSSM whenever mu * tan(beta) is large. In fact, similarly to chargino contributions, gluino contributions are tan(beta) enhanced and can easily dominate charged Higgs contributions for large values of tan(beta). Even though each of the separate contributions to b -> s gamma can be sizeable by itself, surprisingly no absolute lower bound can be placed on any of the relevant SUSY masses, since patterns of partial cancellations among the three competing contributions (Higgs, chargino and gluino) can occur throughout the MSSM parameter space.

hep-ph↗

arXiv2008

Large |Vub|: A challenge for the Minimal Flavour Violating MSSM

Under the assumption of Minimal Flavour Violation (MFV), the Unitarity Triangle (UT) can be determined by using only angle measurements and tree-level observables. In this respect, the most accurate quantities today available are sin 2 beta_{psi Ks}, |Vcb| and |Vub|. Among the latter, |Vub| is at present the quantity suffering from largest systematic uncertainties, given the discrepancy between the inclusive and the exclusive determinations. We show with a numerical fit how sensitively the MFV-UT determination depends on the choice of |Vub|. In addition, we focus on the implications of the inclusive value for |Vub|, which favors two non-SM like solutions in the rhobar - etabar plane. We study in detail the possibility of reproducing such solutions within the MFV MSSM. Our findings indicate that the case for the MFV MSSM is in this respect quite problematic, unless the non-perturbative parameters xi and BK are significantly different from those obtained by lattice methods. As a byproduct, we point out that scenarios with 200 GeV <~ MA <~ 500 GeV and tan beta =~ 50 that predict a significant suppression for Delta Ms in correlation with an enhancement for BR(Bs --> mu^+ mu^-) have to be fine-tuned in order not to violate the new combined bound on the latter decay mode from the CDF and D0 collaborations. Relatively large correlated effects can however still occur for negative values of mu and large values for MA >~ 500 GeV, increasing with increasing tan beta >~ 30.

hep-ph↗