arXiv subjects

James R. Lee

Publications and source records attributed to James R. Lee.

At least 19 recordsLinked to original sources

arXiv2024

Phi-4 Technical Report

We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size -- especially on reasoning-focused benchmarks -- due to improved data, training curriculum, and innovations in the post-training scheme.

cs.CL↗

arXiv2024

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone

We introduce phi-3-mini, a 3.8 billion parameter language model trained on 3.3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite being small enough to be deployed on a phone. Our training dataset is a scaled-up version of the one used for phi-2, composed of heavily filtered publicly available web data and synthetic data. The model is also further aligned for robustness, safety, and chat format. We also provide parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini (e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance multilingual, multimodal, and long-context capabilities, we introduce three models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision. The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters, achieves superior performance in language reasoning, math, and code tasks compared to other open-source models of similar scale, such as Llama 3.1 and the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini. Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from phi-3.5-mini, excels in reasoning tasks and is adept at handling both single-image and text prompts, as well as multi-image and text prompts.

cs.CL↗

arXiv2024

On expanders from the action of GL(2,Z)

Consider the undirected graph $G_n=(V_n, E_n)$ where $V_n = (Z/nZ)^2$ and $E_n$ contains an edge from $(x,y)$ to $(x+1,y)$, $(x,y+1)$, $(x+y,y)$, and $(x,y+x)$ for every $(x,y) \in V_n$. Gabber and Galil, following Margulis, gave an elementary proof that ${G_n}$ forms an expander family. In this note, we present a somewhat simpler proof of this fact, and demonstrate its utility by isolating a key property of the linear transformations $(x,y) -> (x+y,x), (x,y+x)$ that yields expansion. As an example, consider any invertible, integral matrix $S \in GL_2(Z)$ and let $G^S_n = (V_n, E^S_n)$ where $E^S_n$ contains, for every $(x,y) \in V_n$, an edge from $(x,y)$ to $(x+1,y)$, $(x,y+1)$, $S(x,y)$, and $S^T(x,y)$, where $S^T$ denotes the transpose of $S$. Then {G_n^S} forms an expander family if and only if a related infinite graph has positive Cheeger constant. This latter property turns out to be elementary to analyze and can be used to show that {G_n^S} are expanders precisely when the trace of S is non-zero and S is not equal to its transpose. We also present some other generalizations.

math.CO↗

arXiv2023

Sparsifying sums of norms

For any norms $N_1,\ldots,N_m$ on $\mathbb{R}^n$ and $N(x) := N_1(x)+\cdots+N_m(x)$, we show there is a sparsified norm $\tilde{N}(x) = w_1 N_1(x) + \cdots + w_m N_m(x)$ such that $|N(x) - \tilde{N}(x)| \leq εN(x)$ for all $x \in \mathbb{R}^n$, where $w_1,\ldots,w_m$ are non-negative weights, of which only $O(ε^{-2} n \log(n/ε) (\log n)^{2.5} )$ are non-zero. Additionally, if $N$ is $\mathrm{poly}(n)$-equivalent to the Euclidean norm on $\mathbb{R}^n$, then such weights can be found with high probability in time $O(m (\log n)^{O(1)} + \mathrm{poly}(n)) T$, where $T$ is the time required to evaluate a norm $N_i$. This immediately yields analogous statements for sparsifying sums of symmetric submodular functions. More generally, we show how to sparsify sums of $p$th powers of norms when the sum is $p$-uniformly smooth.

cs.DS↗

arXiv2023

Sparsifying generalized linear models

We consider the sparsification of sums $F : \mathbb{R}^n \to \mathbb{R}$ where $F(x) = f_1(\langle a_1,x\rangle) + \cdots + f_m(\langle a_m,x\rangle)$ for vectors $a_1,\ldots,a_m \in \mathbb{R}^n$ and functions $f_1,\ldots,f_m : \mathbb{R} \to \mathbb{R}_+$. We show that $(1+\varepsilon)$-approximate sparsifiers of $F$ with support size $\frac{n}{\varepsilon^2} (\log \frac{n}{\varepsilon})^{O(1)}$ exist whenever the functions $f_1,\ldots,f_m$ are symmetric, monotone, and satisfy natural growth bounds. Additionally, we give efficient algorithms to compute such a sparsifier assuming each $f_i$ can be evaluated efficiently. Our results generalize the classic case of $\ell_p$ sparsification, where $f_i(z) = |z|^p$, for $p \in (0, 2]$, and give the first near-linear size sparsifiers in the well-studied setting of the Huber loss function and its generalizations, e.g., $f_i(z) = \min\{|z|^p, |z|^2\}$ for $0 < p \leq 2$. Our sparsification algorithm can be applied to give near-optimal reductions for optimizing a variety of generalized linear models including $\ell_p$ regression for $p \in (1, 2]$ to high accuracy, via solving $(\log n)^{O(1)}$ sparse regression instances with $m \le n(\log n)^{O(1)}$, plus runtime proportional to the number of nonzero entries in the vectors $a_1, \dots, a_m$.

cs.DS↗

arXiv2022

Spectral hypergraph sparsification via chaining

In a hypergraph on $n$ vertices where $D$ is the maximum size of a hyperedge, there is a weighted hypergraph spectral $\varepsilon$-sparsifier with at most $O(\varepsilon^{-2} \log(D) \cdot n \log n)$ hyperedges. This improves over the bound of Kapralov, Krauthgamer, Tardos and Yoshida (2021) who achieve $O(\varepsilon^{-4} n (\log n)^3)$, as well as the bound $O(\varepsilon^{-2} D^3 n \log n)$ obtained by Bansal, Svensson, and Trevisan (2019). The same sparsification result was obtained independently by Jambulapati, Liu, and Sidford (2022).

math.PR↗

arXiv2021

Multiscale entropic regularization for MTS on general metric spaces

We present an $O((\log n)^2)$-competitive algorithm for metrical task systems (MTS) on any $n$-point metric space that is also $1$-competitive for service costs. This matches the competitive ratio achieved by Bubeck, Cohen, Lee, and Lee (2019) and the refined competitive ratios obtained by Coester and Lee (2019). Those algorithms work by first randomly embedding the metric space into an ultrametric and then solving MTS there. In contrast, our algorithm is cast as regularized gradient descent where the regularizer is a multiscale metric entropy defined directly on the metric space. This answers an open question of Bubeck (Highlights of Algorithms, 2019).

cs.DS↗

arXiv2021

Fusible HSTs and the randomized k-server conjecture

We exhibit an $O((\log k)^6)$-competitive randomized algorithm for the $k$-server problem on any metric space. It is shown that a potential-based algorithm for the fractional $k$-server problem on hierarchically separated trees (HSTs) with competitive ratio $f(k)$ can be used to obtain a randomized algorithm for any metric space with competitive ratio $f(k)^2 O((\log k)^2)$. Employing the $O((\log k)^2)$-competitive algorithm for HSTs from our joint work with Bubeck, Cohen, Lee, and Mądry (2017) yields the claimed bound. The best previous result independent of the geometry of the underlying metric space is the $2k-1$ competitive ratio established for the deterministic work function algorithm by Koutsoupias and Papadimitriou (1995). Even for the special case when the underlying metric space is the real line, the best known competitive ratio was $k$. Since deterministic algorithms can do no better than $k$ on any metric space with at least $k+1$ points, this establishes that for every metric space on which the problem is non-trivial, randomized algorithms give an exponential improvement over deterministic algorithms.

cs.DS↗

arXiv2021

Chemical subdiffusivity of critical 2D percolation

We show that random walk on the incipient infinite cluster (IIC) of two-dimensional critical percolation is subdiffusive in the chemical distance (i.e., in the intrinsic graph metric). Kesten (1986) famously showed that this is true for the Euclidean distance, but it is known that the chemical distance is typically asymptotically larger. More generally, we show that subdiffusivity in the chemical distance holds for stationary random graphs of polynomial volume growth, as long as there is a multi-scale way of covering the graph so that "deep patches" have "thin backbones". Our estimates are quantitative and give explicit bounds in terms of the one and two-arm exponents $η_2 > η_1 > 0$: For $d$-dimensional models, the mean chemical displacement after $T$ steps of random walk scales asymptotically slower than $T^{1/β}$, whenever \[ β< 2 + \frac{η_2-η_1}{d-η_1}\,. \] Using the conjectured values of $η_2 = η_1 + 1/4$ and $η_1 = 5/48$ for 2D lattices, the latter quantity is $2+12/91$.

math.PR↗

arXiv2021

Non-existence of annular separators in geometric graphs

Benjamini and Papasoglou (2011) showed that planar graphs with uniform polynomial volume growth admit $1$-dimensional annular separators: The vertices at graph distance $R$ from any vertex can be separated from those at distance $2R$ by removing at most $O(R)$ vertices. They asked whether geometric $d$-dimensional graphs with uniform polynomial volume growth similarly admit $(d-1)$-dimensional annular separators when $d > 2$. We show that this fails in a strong sense: For any $d \geq 3$ and every $s \geq 1$, there is a collection of interior-disjoint spheres in $\mathbb{R}^d$ whose tangency graph $G$ has uniform polynomial growth, but such that all annular separators in $G$ have cardinality at least $R^s$.

math.CO↗

arXiv2021

Relations between scaling exponents in unimodular random graphs

We investigate the validity of the "Einstein relations" in the general setting of unimodular random networks. These are equalities relating scaling exponents: $d_w = d_f + \tildeζ$ and $d_s = 2 d_f/d_w$, where $d_w$ is the walk dimension, $d_f$ is the fractal dimension, $d_s$ is the spectral dimension, and $\tildeζ$ is the resistance exponent. Roughly speaking, this relates the mean displacement and return probability of a random walker to the density and conductivity of the underlying medium. We show that if $d_f$ and $\tildeζ \geq 0$ exist, then $d_w$ and $d_s$ exist, and the aforementioned equalities hold. Moreover, our primary new estimate is the relation $d_w \geq d_f + \tildeζ$, which is established for all $\tildeζ \in \mathbb{R}$. For the uniform infinite planar triangulation (UIPT), this yields the consequence $d_w=4$ using $d_f=4$ (Angel 2003) and $\tildeζ=0$ (established here as a consequence of the Liouville Quantum Gravity theory, following Gwynne-Miller 2017 and Ding-Gwynne 2020). The conclusion $d_w=4$ had been previously established by Gwynne and Hutchcroft (2018) using more elaborate methods. A new consequence is that $d_w = d_f$ for the uniform infinite Schnyder-wood decorated triangulation, implying that the simple random walk is subdiffusive, since $d_f > 2$ (Ding and Gwynne 2020). For the random walk on $\mathbb{Z}^2$ driven by conductances from an exponentiated Gaussian free field with exponent $γ> 0$, one has $d_f = d_f(γ)$ and $\tildeζ=0$ (Biskup, Ding, and Goswami 2020). This yields $d_s=2$ and $d_w = d_f$, confirming two predictions of those authors.

math.PR↗

arXiv2021

On planar graphs of uniform polynomial growth

Consider an infinite planar graph with uniform polynomial growth of degree d > 2. Many examples of such graphs exhibit similar geometric and spectral properties, and it has been conjectured that this is necessary. We present a family of counterexamples. In particular, we show that for every rational d > 2, there is a planar graph with uniform polynomial growth of degree d on which the random walk is transient, disproving a conjecture of Benjamini (2011). By a well-known theorem of Benjamini and Schramm, such a graph cannot be a unimodular random graph. We also give examples of unimodular random planar graphs of uniform polynomial growth with unexpected properties. For instance, graphs of (almost sure) uniform polynomial growth of every rational degree d > 2 for which the speed exponent of the walk is larger than 1/d, and in which the complements of all balls are connected. This resolves negatively two questions of Benjamini and Papasoglou (2011).

math.PR↗

arXiv2020

Metrical task systems on trees via mirror descent and unfair gluing

We consider metrical task systems on tree metrics, and present an $O(\mathrm{depth} \times \log n)$-competitive randomized algorithm based on the mirror descent framework introduced in our prior work on the $k$-server problem. For the special case of hierarchically separated trees (HSTs), we use mirror descent to refine the standard approach based on gluing unfair metrical task systems. This yields an $O(\log n)$-competitive algorithm for HSTs, thus removing an extraneous $\log\log n$ in the bound of Fiat and Mendel (2003). Combined with well-known HST embedding theorems, this also gives an $O((\log n)^2)$-competitive randomized algorithm for every $n$-point metric space.

cs.DS↗

arXiv2020

Pure entropic regularization for metrical task systems

We show that on every $n$-point HST metric, there is a randomized online algorithm for metrical task systems (MTS) that is $1$-competitive for service costs and $O(\log n)$-competitive for movement costs. In general, these refined guarantees are optimal up to the implicit constant. While an $O(\log n)$-competitive algorithm for MTS on HST metrics was developed by Bubeck et al. (SODA 2019), that approach could only establish an $O((\log n)^2)$-competitive ratio when the service costs are required to be $O(1)$-competitive. Our algorithm can be viewed as an instantiation of online mirror descent with the regularizer derived from a multiscale conditional entropy. In fact, our algorithm satisfies a set of even more refined guarantees; we are able to exploit this property to combine it with known random embedding theorems and obtain, for any $n$-point metric space, a randomized algorithm that is $1$-competitive for service costs and $O((\log n)^2)$-competitive for movement costs.

cs.DS↗

arXiv2020

Conformal growth rates and spectral geometry on distributional limits of graphs

For a unimodular random graph $(G,ρ)$, we consider deformations of its intrinsic path metric by a (random) weighting of its vertices. This leads to the notion of the conformal growth exponent of $(G,ρ)$, which is the best asymptotic degree of volume growth of balls that can be achieved by such a reweighting. Under moment conditions on the degree of the root, we show that the conformal growth exponent of a unimodular random graph bounds its almost sure spectral dimension. This has interesting consequences for many low-dimensional models. The consequences in dimension two are particularly strong. It establishes that models like the uniform infinite planar triangulation (UIPT) and quadrangulation (UIPQ) almost surely have spectral dimension at most two. It also establishes a conjecture of Benjamini and Schramm (2001) by extending their Recurrence Theorem from planar graphs to arbitrary families of $H$-minor free graphs. More generally, it strengthens the work of Gurel-Gurevich and Nachmias (2013) who established recurrence for distributional limits of planar graphs when the degree of the root has exponential tails. We further present a general method for proving subdiffusivity of the random walk on a large class of models, including UIPT and UIPQ, using only the volume growth profile of balls in the intrinsic metric.

math.PR↗

arXiv2020

Adversarial hypothesis testing and a quantum Stein's Lemma for restricted measurements

Recall the classical hypothesis testing setting with two convex sets of probability distributions P and Q. One receives either n i.i.d. samples from a distribution p in P or from a distribution q in Q and wants to decide from which set the points were sampled. It is known that the optimal exponential rate at which errors decrease can be achieved by a simple maximum-likelihood ratio test which does not depend on p or q, but only on the sets P and Q. We consider an adaptive generalization of this model where the choice of p in P and q in Q can change in each sample in some way that depends arbitrarily on the previous samples. In other words, in the k'th round, an adversary, having observed all the previous samples in rounds 1,...,k-1, chooses p_k in P and q_k in Q, with the goal of confusing the hypothesis test. We prove that even in this case, the optimal exponential error rate can be achieved by a simple maximum-likelihood test that depends only on P and Q. We then show that the adversarial model has applications in hypothesis testing for quantum states using restricted measurements. For example, it can be used to study the problem of distinguishing entangled states from the set of all separable states using only measurements that can be implemented with local operations and classical communication (LOCC). The basic idea is that in our setup, the deleterious effects of entanglement can be simulated by an adaptive classical adversary. We prove a quantum Stein's Lemma in this setting: In many circumstances, the optimal hypothesis testing rate is equal to an appropriate notion of quantum relative entropy between two states. In particular, our arguments yield an alternate proof of Li and Winter's recent strengthening of strong subadditivity for quantum relative entropy.

cs.IT↗

arXiv2018

Flow-Cut Gaps and Face Covers in Planar Graphs

The relationship between the sparsest cut and the maximum concurrent multi-flow in graphs has been studied extensively. For general graphs with $k$ terminal pairs, the flow-cut gap is $O(\log k)$, and this is tight. But when topological restrictions are placed on the flow network, the situation is far less clear. In particular, it has been conjectured that the flow-cut gap in planar networks is $O(1)$, while the known bounds place the gap somewhere between $2$ (Lee and Raghavendra, 2003) and $O(\sqrt{\log k})$ (Rao, 1999). A seminal result of Okamura and Seymour (1981) shows that when all the terminals of a planar network lie on a single face, the flow-cut gap is exactly $1$. This setting can be generalized by considering planar networks where the terminals lie on $γ>1$ faces in some fixed planar drawing. Lee and Sidiropoulos (2009) proved that the flow-cut gap is bounded by a function of $γ$, and Chekuri, Shepherd, and Weibel (2013) showed that the gap is at most $3γ$. We prove that the flow-cut gap is $O(\logγ)$, by showing that the edge-weighted shortest-path metric induced on the terminals admits a stochastic embedding into trees with distortion $O(\logγ)$, which is tight. The preceding results refer to the setting of edge-capacitated networks. For vertex-capacitated networks, it can be significantly more challenging to control flow-cut gaps. While there is no exact vertex-capacitated version of the Okamura-Seymour Theorem, an approximate version holds; Lee, Mendel, and Moharrami (2015) showed that the vertex-capacitated flow-cut gap is $O(1)$ on planar networks whose terminals lie on a single face. We prove that the flow-cut gap is $O(γ)$ for vertex-capacitated instances when the terminals lie on at most $γ$ faces. In fact, this result holds in the more general setting of submodular vertex capacities.

cs.DS↗

arXiv2018

Discrete uniformizing metrics on distributional limits of sphere packings

Suppose that $\{G_n\}$ is a sequence of finite graphs such that each $G_n$ is the tangency graph of a sphere packing in $\mathbb{R}^d$. Let $ρ_n$ be a uniformly random vertex of $G_n$ and suppose that $(G,ρ)$ is the distributional limit of $\{(G_n,ρ_n)\}$ in the sense of Benjamini and Schramm. Then the conformal growth exponent of $(G,ρ)$ is at most $d$. In other words, there exists a unimodular "unit volume" weighting of the graph metric on $(G,ρ)$ such that the volume growth of balls in the weighted path metric is bounded by a polynomial of degree $d$. This generalizes to limits of graphs that can be "coarsely" packed in an Ahlfors $d$-regular metric measure space. Using our previous work, this implies that, under moment conditions on the degree of the root $ρ$,the almost sure spectral dimension of $G$ is at most $d$. This fact was known previously only for graphs packed in $\mathbb{R}^2$ (planar graphs), and the case of $d > 2$ eluded approaches based on extremal length. In the process of bounding the spectral dimension, we establish that the spectral measure of $(G,ρ)$ is dominated by a variant of the $d$-dimensional Weyl law.

math.MG↗