arXiv subjects

Jalaj Upadhyay

Publications and source records attributed to Jalaj Upadhyay.

29 records · Page 2Linked to original sources

arXiv2023

Differentially Private Sampling from Rashomon Sets, and the Universality of Langevin Diffusion for Convex Optimization

In this paper we provide an algorithmic framework based on Langevin diffusion (LD) and its corresponding discretizations that allow us to simultaneously obtain: i) An algorithm for sampling from the exponential mechanism, whose privacy analysis does not depend on convexity and which can be stopped at anytime without compromising privacy, and ii) tight uniform stability guarantees for the exponential mechanism. As a direct consequence, we obtain optimal excess empirical and population risk guarantees for (strongly) convex losses under both pure and approximate differential privacy (DP). The framework allows us to design a DP uniform sampler from the Rashomon set. Rashomon sets are widely used in interpretable and robust machine learning, understanding variable importance, and characterizing fairness.

cs.LG↗

arXiv2023

A Unifying Framework for Differentially Private Sums under Continual Observation

We study the problem of maintaining a differentially private decaying sum under continual observation. We give a unifying framework and an efficient algorithm for this problem for \emph{any sufficiently smooth} function. Our algorithm is the first differentially private algorithm that does not have a multiplicative error for polynomially-decaying weights. Our algorithm improves on all prior works on differentially private decaying sums under continual observation and recovers exactly the additive error for the special case of continual counting from Henzinger et al. (SODA 2023) as a corollary. Our algorithm is a variant of the factorization mechanism whose error depends on the $γ_2$ and $γ_F$ norm of the underlying matrix. We give a constructive proof for an almost exact upper bound on the $γ_2$ and $γ_F$ norm and an almost tight lower bound on the $γ_2$ norm for a large class of lower-triangular matrices. This is the first non-trivial lower bound for lower-triangular matrices whose non-zero entries are not all the same. It includes matrices for all continual decaying sums problems, resulting in an upper bound on the additive error of any differentially private decaying sums algorithm under continual observation. We also explore some implications of our result in discrepancy theory and operator algebra. Given the importance of the $γ_2$ norm in computer science and the extensive work in mathematics, we believe our result will have further applications.

cs.LG↗

arXiv2023

Near Optimal Linear Algebra in the Online and Sliding Window Models

We initiate the study of numerical linear algebra in the sliding window model, where only the most recent $W$ updates in a stream form the underlying data set. We first introduce a unified row-sampling based framework that gives randomized algorithms for spectral approximation, low-rank approximation/projection-cost preservation, and $\ell_1$-subspace embeddings in the sliding window model, which often use nearly optimal space and achieve nearly input sparsity runtime. Our algorithms are based on "reverse online" versions of offline sampling distributions such as (ridge) leverage scores, $\ell_1$ sensitivities, and Lewis weights to quantify both the importance and the recency of a row. Our row-sampling framework rather surprisingly implies connections to the well-studied online model; our structural results also give the first sample optimal (up to lower order terms) online algorithm for low-rank approximation/projection-cost preservation. Using this powerful primitive, we give online algorithms for column/row subset selection and principal component analysis that resolves the main open question of Bhaskara et. al.,(FOCS 2019). We also give the first online algorithm for $\ell_1$-subspace embeddings. We further formalize the connection between the online model and the sliding window model by introducing an additional unified framework for deterministic algorithms using a merge and reduce paradigm and the concept of online coresets. Our sampling based algorithms in the row-arrival online model yield online coresets, giving deterministic algorithms for spectral approximation, low-rank approximation/projection-cost preservation, and $\ell_1$-subspace embeddings in the sliding window model that use nearly optimal space.

cs.DS↗

arXiv2023

Differentially Private Range Query on Shortest Paths

We consider differentially private range queries on a graph where query ranges are defined as the set of edges on a shortest path of the graph. Edges in the graph carry sensitive attributes and the goal is to report the sum of these attributes on a shortest path for counting query or the minimum of the attributes in a bottleneck query. We use differential privacy to ensure that the release of these query answers provide protection of the privacy of the sensitive edge attributes. Our goal is to develop mechanisms that minimize the additive error of the reported answers with the given privacy budget. In this paper we report non-trivial results for private range queries on shortest paths. For counting range queries we can achieve an additive error of $\tilde O(n^{1/3})$ for $\varepsilon$-DP and $\tilde O(n^{1/4})$ for $(\varepsilon, δ)$-DP. We present two algorithms where we control the final error by carefully balancing perturbation added to the edge attributes directly versus perturbation added to a subset of range query answers (which can be used for other range queries). Bottleneck range queries are easier and can be answered with polylogarithmic additive errors using standard techniques.

cs.DS↗

arXiv2020

A Framework for Private Matrix Analysis

We study private matrix analysis in the sliding window model where only the last $W$ updates to matrices are considered useful for analysis. We give first efficient $o(W)$ space differentially private algorithms for spectral approximation, principal component analysis, and linear regression. We also initiate and show efficient differentially private algorithms for two important variants of principal component analysis: sparse principal component analysis and non-negative principal component analysis. Prior to our work, no such result was known for sparse and non-negative differentially private principal component analysis even in the static data setting. These algorithms are obtained by identifying sufficient conditions on positive semidefinite matrices formed from streamed matrices. We also show a lower bound on space required to compute low-rank approximation even if the algorithm gives multiplicative approximation and incurs additive error. This follows via reduction to a certain communication complexity problem.

cs.LG↗

arXiv2018

The Price of Differential Privacy for Low-Rank Factorization

In this paper, we study what price one has to pay to release {\em differentially private low-rank factorization} of a matrix. We consider various settings that are close to the real world applications of low-rank factorization: (i) the manner in which matrices are updated (row by row or in an arbitrary manner), (ii) whether matrices are distributed or not, and (iii) how the output is produced (once at the end of all updates, also known as {\em one-shot algorithms} or continually). Even though these settings are well studied without privacy, surprisingly, there are no private algorithm for these settings (except when a matrix is updated row by row). We present the first set of differentially private algorithms for all these settings. Our algorithms when private matrix is updated in an arbitrary manner promise differential privacy with respect to two stronger privacy guarantees than previously studied, use space and time {\em comparable} to the non-private algorithm, and achieve {\em optimal accuracy}. To complement our positive results, we also prove that the space required by our algorithms is optimal up to logarithmic factors. When data matrices are distributed over multiple servers, we give a non-interactive differentially private algorithm with communication cost independent of dimension. In concise, we give algorithms that incur optimal cost. We also perform experiments to verify that all our algorithms perform well in practice and outperform the best known algorithms until now for large range of parameters. We give experimental results for total approximation error and additive error for varying dimensions, $α$ and $k$.

cs.DS↗

arXiv2017

Differentially Private Linear Algebra in the Streaming Model

Numerical linear algebra plays an important role in computer science. In this paper, we initiate the study of performing linear algebraic tasks while preserving privacy when the data is streamed online. Our main focus is the space requirement of the privacy-preserving data-structures. We give the first {\em sketch-based} algorithm for differential privacy. We give optimal, up to logarithmic factor, space data-structures that can compute low rank approximation, linear regression, and matrix multiplication, while preserving differential privacy with better additive error bounds compared to the known results. Notably, we match the best known space bound in the non-private setting by Kane and Nelson (J. ACM, 61(1):4). Our mechanism for differentially private low-rank approximation {\em reuses} the random Gaussian matrix in a specific way to provide a single-pass mechanism. We prove that the resulting distribution also preserve differential privacy. This can be of independent interest. We do not make any assumptions, like singular value separation or normalized row assumption, as made in the earlier works. The mechanisms for matrix multiplication and linear regression can be seen as the private analogues of the known non-private algorithms. All our mechanisms, in the form presented, can also be computed in the distributed setting.

cs.DS↗

arXiv2016

On Low-Space Differentially Private Low-rank Factorization in the Spectral Norm

Low-rank factorization is used in many areas of computer science where one performs spectral analysis on large sensitive data stored in the form of matrices. In this paper, we study differentially private low-rank factorization of a matrix with respect to the spectral norm in the turnstile update model. In this problem, given an input matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$ updated in the turnstile manner and a target rank $k$, the goal is to find two rank-$k$ orthogonal matrices $\mathbf{U}_k \in \mathbb{R}^{m \times k}$ and $\mathbf{V}_k \in \mathbb{R}^{n \times k}$, and one positive semidefinite diagonal matrix $\textbfΣ_k \in \mathbb{R}^{k \times k}$ such that $\mathbf{A} \approx \mathbf{U}_k \textbfΣ_k \mathbf{V}_k^\mathsf{T}$ with respect to the spectral norm. Our main contributions are two computationally efficient and sub-linear space algorithms for computing a differentially private low-rank factorization. We consider two levels of privacy. In the first level of privacy, we consider two matrices neighboring if their difference has a Frobenius norm at most $1$. In the second level of privacy, we consider two matrices as neighboring if their difference can be represented as an outer product of two unit vectors. Both these privacy levels are stronger than those studied in the earlier papers such as Dwork {\it et al.} (STOC 2014), Hardt and Roth (STOC 2013), and Hardt and Price (NIPS 2014). As a corollary to our results, we get non-private algorithms that compute low-rank factorization in the turnstile update model with respect to the spectral norm. We note that, prior to this work, no algorithm that outputs low-rank factorization with respect to the spectral norm in the turnstile update model was known; i.e., our algorithm gives the first non-private low-rank factorization with respect to the spectral norm in the turnstile update mode.

cs.DS↗

arXiv2016

Multi-prover Proof-of-Retrievability

There has been considerable recent interest in "cloud storage" wherein a user asks a server to store a large file. One issue is whether the user can verify that the server is actually storing the file, and typically a challenge-response protocol is employed to convince the user that the file is indeed being stored correctly. The security of these schemes is phrased in terms of an extractor which will recover the file given any "proving algorithm" that has a sufficiently high success probability. This forms the basis of \emph{proof-of-retrievability} ($\mathsf{PoR}$) systems. In this paper, we study multiple server $\mathsf{PoR}$ systems. We formalize security definitions for two possible scenarios: (i) when a threshold of servers succeed with high enough probability (worst-case) and (ii) when the average of the success probability of all the servers is above a threshold (average-case). We also motivate the study of confidentiality of the outsourced message. We give $\mathsf{M}\mbox{-}\mathsf{PoR}$ schemes which are secure under both these security definitions and provide reasonable confidentiality guarantees even when there is no restriction on the computational power of the servers. We also show how classical statistical techniques used by Paterson, Stinson and Upadhyay (Journal of Mathematical Cryptology: 7(3)) can be extended to evaluate whether the responses of the provers are accurate enough to permit successful extraction. We also look at one specific instantiation of our construction when instantiated with the unconditionally secure version of the Shacham-Waters scheme (Asiacrypt, 2008). This scheme gives reasonable security and privacy guarantee. We show that, in the multi-server setting with computationally unbounded provers, one can overcome the limitation that the verifier needs to store as much secret information as the provers.

cs.CR↗

arXiv2015

Randomness Efficient Fast-Johnson-Lindenstrauss Transform with Applications in Differential Privacy and Compressed Sensing

The Johnson-Lindenstrauss property ({\sf JLP}) of random matrices has immense application in computer science ranging from compressed sensing, learning theory, numerical linear algebra, to privacy. This paper explores the properties and applications of a distribution of random matrices. Our distribution satisfies {\sf JLP} with desirable properties like fast matrix-vector multiplication, sparsity, and optimal subspace embedding. We can sample a random matrix from this distribution using exactly $2n+n \log n$ random bits. We show that a random matrix picked from this distribution preserves differential privacy under the condition that the input private matrix satisfies certain spectral property. This improves the run-time of various differentially private mechanisms like Blocki {\it et al.} (FOCS 2012) and Upadhyay (ASIACRYPT 13). Our final construction has a bounded column sparsity. Therefore, this answers an open problem stated in Blocki {\it et al.} (FOCS 2012). Using the results of Baranuik {\it et al.} (Constructive Approximation: 28(3)), our result implies a randomness efficient matrices that satisfies the Restricted-Isometry Property of optimal order for small sparsity with exactly linear random bits. We also show that other known distributions of sparse random matrices with the {\sf JLP} does not preserves differential privacy; thereby, answering one of the open problem posed by Blocki {\it et al.} (FOCS 2012). Extending on the works of Kane and Nelson (JACM: 61(1)), we also give unified analysis of some of the known Johnson-Lindenstrauss transform. We also present a self-contained simplified proof of an inequality on quadratic form of Gaussian variables that we use in all our proofs.

cs.DS↗

arXiv2012

A coding theory foundation for the analysis of general unconditionally secure proof-of-retrievability schemes for cloud storage

There has been considerable recent interest in "cloud storage" wherein a user asks a server to store a large file. One issue is whether the user can verify that the server is actually storing the file, and typically a challenge-response protocol is employed to convince the user that the file is indeed being stored correctly. The security of these schemes is phrased in terms of an extractor which will recover or retrieve the file given any "proving algorithm" that has a sufficiently high success probability. This paper treats proof-of-retrievability schemes in the model of unconditional security, where an adversary has unlimited computational power. In this case retrievability of the file can be modelled as error-correction in a certain code. We provide a general analytical framework for such schemes that yields exact (non-asymptotic) reductions that precisely quantify conditions for extraction to succeed as a function of the success probability of a proving algorithm, and we apply this analysis to several archetypal schemes. In addition, we provide a new methodology for the analysis of keyed POR schemes in an unconditionally secure setting, and use it to prove the security of a modified version of a scheme due to Shacham and Waters under a slightly restricted attack model, thus providing the first example of a keyed POR scheme with unconditional security. We also show how classical statistical techniques can be used to evaluate whether the responses of the prover are accurate enough to permit successful extraction. Finally, we prove a new lower bound on storage and communication complexity of POR schemes.

cs.CR↗