arXiv subjects

W. Ross Morrow

Publications and source records attributed to W. Ross Morrow.

5 recordsLinked to original sources

arXiv2026

Multi-Head Self Attention is a Parameter Identification Mechanism

We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to the total number of parameters scales like the reciprocal of the number of heads ($1/2 \to 1/(2H)$), meaning models with more heads are structurally more identified. A subtle side effect of the mathematics observation that attention can never be fully identified. Similarly we also show that some bias terms can have no effect on softmax-based attention layers in both the single- and multiple-head settings, though this is mostly a curiosity that should have a marginal effect on model size and model training/prediction efficiency. We also touch on modern improvements to transformers including RoPE and GQA from this perspective, illustrating how those as well can improve the ratio of ``meaningful'' parameters to all parameters. Simple numerical examples demonstrate that training can indeed involve updates that overlap model-invariant subspaces that arise from a lack of identification. As part of our experiments we use a ``rebalancing'' approach that can ``fix'' updates that overlap unindentified subspaces but do not try to present evidence this should actually be adopted. Instead we simply view our numerical results as exploring and confirming the theoretical results. As a whole we discuss a purely mathematical/statistical explanation, identification, for why specific architectural choices in transformers may have improved performance.

cs.LG↗

arXiv2015

Should Optimal Designers Worry About Consideration?

Consideration set formation using non-compensatory screening rules is a vital component of real purchasing decisions with decades of experimental validation. Marketers have recently developed statistical methods that can estimate quantitative choice models that include consideration set formation via non-compensatory screening rules. But is capturing consideration within models of choice important for design? This paper reports on a simulation study of a vehicle portfolio design when households screen over vehicle body style built to explore the importance of capturing consideration rules for optimal designers. We generate synthetic market share data, fit a variety of discrete choice models to the data, and then optimize design decisions using the estimated models. Model predictive power, design "error", and profitability relative to ideal profits are compared as the amount of market data available increases. We find that even when estimated compensatory models provide relatively good predictive accuracy, they can lead to sub-optimal design decisions when the population uses consideration behavior; convergence of compensatory models to non-compensatory behavior is likely to require unrealistic amounts of data; and modeling heterogeneity in non-compensatory screening is more valuable than heterogeneity in compensatory trade-offs. This supports the claim that designers should carefully identify consideration behaviors before optimizing product portfolios. We also find that higher model predictive power does not necessarily imply better design decisions; that is, different model forms can provide "descriptive" rather than "predictive" information that is useful for design.

stat.AP↗

arXiv2012

On the Existence of Bertrand-Nash Equilibrium Prices Under Logit Demand

This article presents a proof of the existence of Bertrand-Nash equilibrium prices with multi-product firms and under the Logit model of demand that does not rely on restrictive assumptions on product characteristics, firm homogeneity or symmetry, product costs, or linearity of the utility function. The proof is based on conditions for the indirect utility function, fixed-point equations derived from the first-order conditions, and a direct analysis of the second-order conditions resulting in the uniqueness of profit-maximizing prices. Several subsequent results also demonstrate that price equilibrium under the Logit model of demand cannot adequately describe multi-product pricing.

q-fin.GN↗

arXiv2011

Hessian-Free Methods for Checking the Second-Order Sufficient Conditions in Equality-Constrained Optimization and Equilibrium Problems

Verifying the Second-Order Sufficient Condition (SOSC), thus ensuring a stationary point locally minimizes a given objective function (subject to certain constraints), is an essential component of non-convex computational optimization and equilibrium programming. This article proposes three new "Hessian-free" tests of the SOSC that can be implemented efficiently with gradient evaluations alone and reveal feasible directions of negative curvature when the SOSC fails. The Bordered Hessian Test and a Matrix Inertia test, two classical tests of the SOSC, require explicit knowledge of the Hessian of the Lagrangian and do not reveal feasible directions of negative curvature should the SOSC fail. Computational comparisons of the new methods with classical tests demonstrate the relative efficiency of these new algorithms and the need for careful study of false negatives resulting from accumulation of round-off errors.

math.OC↗

arXiv2010

Fixed-Point Approaches to Computing Bertrand-Nash Equilibrium Prices Under Mixed Logit Demand: A Technical Framework for Analysis and Efficient Computational Methods

This article presents a detailed technical framework for modeling with Bertrand-Nash equilibrium prices under Mixed Logit demand. Two coercive fixed-point equations provide more stable computational methods than those obtained from the literal first-order conditions. Assumptions to justify derivation and use of these equations are provided. A brief discussion of a GMRES-Newton method with hookstep globalization strategy originally due to Viswanath is also given. This article can be considered a supplement to an article by the authors forthcoming in the journal {\em Operations Research}.

math.NA↗