arXiv subjects

Mihailo Stojnic

Publications and source records attributed to Mihailo Stojnic.

At least 73 records · Page 4Linked to original sources

arXiv2015

Upper-bounding $\ell_1$-optimization sectional thresholds

In this paper we look at a particular problem related to under-determined linear systems of equations with sparse solutions. $\ell_1$-minimization is a fairly successful polynomial technique that can in certain statistical scenarios find sparse enough solutions of such systems. Barriers of $\ell_1$ performance are typically referred to as its thresholds. Depending if one is interested in a typical or worst case behavior one then distinguishes between the \emph{weak} thresholds that relate to a typical behavior on one side and the \emph{sectional} and \emph{strong} thresholds that relate to the worst case behavior on the other side. Starting with seminal works \cite{CRT,DonohoPol,DOnoho06CS} a substantial progress has been achieved in theoretical characterization of $\ell_1$-minimization statistical thresholds. More precisely, \cite{CRT,DOnoho06CS} presented for the first time linear lower bounds on all of these thresholds. Donoho's work \cite{DonohoPol} (and our own \cite{StojnicCSetam09,StojnicUpper10}) went a bit further and essentially settled the $\ell_1$'s \emph{weak} thresholds. At the same time they also provided fairly good lower bounds on the values on the \emph{sectional} and \emph{strong} thresholds. In this paper, we revisit the \emph{sectional} thresholds and present a simple mechanism that can be used to create solid upper bounds as well. The method we present relies on a seemingly simple but substantial progress we made in studying Hopfield models in \cite{StojnicHopBnds10}.

cs.IT↗

arXiv2015

Bounds on restricted isometry constants of random matrices

In this paper we look at isometry properties of random matrices. During the last decade these properties gained a lot attention in a field called compressed sensing in first place due to their initial use in \cite{CRT,CT}. Namely, in \cite{CRT,CT} these quantities were used as a critical tool in providing a rigorous analysis of $\ell_1$ optimization's ability to solve an under-determined system of linear equations with sparse solutions. In such a framework a particular type of isometry, called restricted isometry, plays a key role. One then typically introduces a couple of quantities, called upper and lower restricted isometry constants to characterize the isometry properties of random matrices. Those constants are then usually viewed as mathematical objects of interest and their a precise characterization is desirable. The first estimates of these quantities within compressed sensing were given in \cite{CRT,CT}. As the need for precisely estimating them grew further a finer improvements of these initial estimates were obtained in e.g. \cite{BCTsharp09,BT10}. These are typically obtained through a combination of union-bounding strategy and powerful tail estimates of extreme eigenvalues of Wishart (Gaussian) matrices (see, e.g. \cite{Edelman88}). In this paper we attempt to circumvent such an approach and provide an alternative way to obtain similar estimates.

math.OC↗

arXiv2015

Towards a better compressed sensing

In this paper we look at a well known linear inverse problem that is one of the mathematical cornerstones of the compressed sensing field. In seminal works \cite{CRT,DOnoho06CS} $\ell_1$ optimization and its success when used for recovering sparse solutions of linear inverse problems was considered. Moreover, \cite{CRT,DOnoho06CS} established for the first time in a statistical context that an unknown vector of linear sparsity can be recovered as a known existing solution of an under-determined linear system through $\ell_1$ optimization. In \cite{DonohoPol,DonohoUnsigned} (and later in \cite{StojnicCSetam09,StojnicUpper10}) the precise values of the linear proportionality were established as well. While the typical $\ell_1$ optimization behavior has been essentially settled through the work of \cite{DonohoPol,DonohoUnsigned,StojnicCSetam09,StojnicUpper10}, we in this paper look at possible upgrades of $\ell_1$ optimization. Namely, we look at a couple of algorithms that turn out to be capable of recovering a substantially higher sparsity than the $\ell_1$. However, these algorithms assume a bit of "feedback" to be able to work at full strength. This in turn then translates the original problem of improving upon $\ell_1$ to designing algorithms that would be able to provide output needed to feed the $\ell_1$ upgrades considered in this papers.

cs.IT↗

arXiv2015

Compressed sensing of block-sparse positive vectors

In this paper we revisit one of the classical problems of compressed sensing. Namely, we consider linear under-determined systems with sparse solutions. A substantial success in mathematical characterization of an $\ell_1$ optimization technique typically used for solving such systems has been achieved during the last decade. Seminal works \cite{CRT,DOnoho06CS} showed that the $\ell_1$ can recover a so-called linear sparsity (i.e. solve systems even when the solution has a sparsity linearly proportional to the length of the unknown vector). Later considerations \cite{DonohoPol,DonohoUnsigned} (as well as our own ones \cite{StojnicCSetam09,StojnicUpper10}) provided the precise characterization of this linearity. In this paper we consider the so-called structured version of the above sparsity driven problem. Namely, we view a special case of sparse solutions, the so-called block-sparse solutions. Typically one employs $\ell_2/\ell_1$-optimization as a variant of the standard $\ell_1$ to handle block-sparse case of sparse solution systems. We considered systems with block-sparse solutions in a series of work \cite{StojnicCSetamBlock09,StojnicUpperBlock10,StojnicICASSP09block,StojnicJSTSP09} where we were able to provide precise performance characterizations if the $\ell_2/\ell_1$-optimization similar to those obtained for the standard $\ell_1$ optimization in \cite{StojnicCSetam09,StojnicUpper10}. Here we look at a similar class of systems where on top of being block-sparse the unknown vectors are also known to have components of the same sign. In this paper we slightly adjust $\ell_2/\ell_1$-optimization to account for the known signs and provide a precise performance characterization of such an adjustment.

cs.IT↗

arXiv2013

Bounding ground state energy of Hopfield models

In this paper we look at a class of random optimization problems that arise in the forms typically known as Hopfield models. We view two scenarios which we term as the positive Hopfield form and the negative Hopfield form. For both of these scenarios we define the binary optimization problems that essentially emulate what would typically be known as the ground state energy of these models. We then present a simple mechanism that can be used to create a set of theoretical rigorous bounds for these energies. In addition to purely theoretical bounds, we also present a couple of fast optimization algorithms that can also be used to provide solid (albeit a bit weaker) algorithmic bounds for the ground state energies.

math.OC↗

arXiv2013

Lifting $\ell_1$-optimization strong and sectional thresholds

In this paper we revisit under-determined linear systems of equations with sparse solutions. As is well known, these systems are among core mathematical problems of a very popular compressed sensing field. The popularity of the field as well as a substantial academic interest in linear systems with sparse solutions are in a significant part due to seminal results \cite{CRT,DonohoPol}. Namely, working in a statistical scenario, \cite{CRT,DonohoPol} provided substantial mathematical progress in characterizing relation between the dimensions of the systems and the sparsity of unknown vectors recoverable through a particular polynomial technique called $\ell_1$-minimization. In our own series of work \cite{StojnicCSetam09,StojnicUpper10,StojnicEquiv10} we also provided a collection of mathematical results related to these problems. While, Donoho's work \cite{DonohoPol,DonohoUnsigned} established (and our own work \cite{StojnicCSetam09,StojnicUpper10,StojnicEquiv10} reaffirmed) the typical or the so-called \emph{weak threshold} behavior of $\ell_1$-minimization many important questions remain unanswered. Among the most important ones are those that relate to non-typical or the so-called \emph{strong threshold} behavior. These questions are usually combinatorial in nature and known techniques come up short of providing the exact answers. In this paper we provide a powerful mechanism that that can be used to attack the "tough" scenario, i.e. the \emph{strong threshold} (and its a similar form called \emph{sectional threshold}) of $\ell_1$-minimization.

cs.IT↗

arXiv2013

Under-determined linear systems and $\ell_q$-optimization thresholds

Recent studies of under-determined linear systems of equations with sparse solutions showed a great practical and theoretical efficiency of a particular technique called $\ell_1$-optimization. Seminal works \cite{CRT,DOnoho06CS} rigorously confirmed it for the first time. Namely, \cite{CRT,DOnoho06CS} showed, in a statistical context, that $\ell_1$ technique can recover sparse solutions of under-determined systems even when the sparsity is linearly proportional to the dimension of the system. A followup \cite{DonohoPol} then precisely characterized such a linearity through a geometric approach and a series of work\cite{StojnicCSetam09,StojnicUpper10,StojnicEquiv10} reaffirmed statements of \cite{DonohoPol} through a purely probabilistic approach. A theoretically interesting alternative to $\ell_1$ is a more general version called $\ell_q$ (with an essentially arbitrary $q$). While $\ell_1$ is typically considered as a first available convex relaxation of sparsity norm $\ell_0$, $\ell_q,0\leq q\leq 1$, albeit non-convex, should technically be a tighter relaxation of $\ell_0$. Even though developing polynomial (or close to be polynomial) algorithms for non-convex problems is still in its initial phases one may wonder what would be the limits of an $\ell_q,0\leq q\leq 1$, relaxation even if at some point one can develop algorithms that could handle its non-convexity. A collection of answers to this and a few realted questions is precisely what we present in this paper.

cs.IT↗

arXiv2013

Spherical perceptron as a storage memory with limited errors

It has been known for a long time that the classical spherical perceptrons can be used as storage memories. Seminal work of Gardner, \cite{Gar88}, started an analytical study of perceptrons storage abilities. Many of the Gardner's predictions obtained through statistical mechanics tools have been rigorously justified. Among the most important ones are of course the storage capacities. The first rigorous confirmations were obtained in \cite{SchTir02,SchTir03} for the storage capacity of the so-called positive spherical perceptron. These were later reestablished in \cite{TalBook} and a bit more recently in \cite{StojnicGardGen13}. In this paper we consider a variant of the spherical perceptron that operates as a storage memory but allows for a certain fraction of errors. In Gardner's original work the statistical mechanics predictions in this directions were presented sa well. Here, through a mathematically rigorous analysis, we confirm that the Gardner's predictions in this direction are in fact provable upper bounds on the true values of the storage capacity. Moreover, we then present a mechanism that can be used to lower these bounds. Numerical results that we present indicate that the Garnder's storage capacity predictions may, in a fairly wide range of parameters, be not that far away from the true values.

math.PR↗

arXiv2013

Lifting/lowering Hopfield models ground state energies

In our recent work \cite{StojnicHopBnds10} we looked at a class of random optimization problems that arise in the forms typically known as Hopfield models. We viewed two scenarios which we termed as the positive Hopfield form and the negative Hopfield form. For both of these scenarios we defined the binary optimization problems whose optimal values essentially emulate what would typically be known as the ground state energy of these models. We then presented a simple mechanisms that can be used to create a set of theoretical rigorous bounds for these energies. In this paper we create a way more powerful set of mechanisms that can substantially improve the simple bounds given in \cite{StojnicHopBnds10}. In fact, the mechanisms we create in this paper are the first set of results that show that convexity type of bounds can be substantially improved in this type of combinatorial problems.

math.OC↗

arXiv2013

Lifting $\ell_q$-optimization thresholds

In this paper we look at a connection between the $\ell_q,0\leq q\leq 1$, optimization and under-determined linear systems of equations with sparse solutions. The case $q=1$, or in other words $\ell_1$ optimization and its a connection with linear systems has been thoroughly studied in last several decades; in fact, especially so during the last decade after the seminal works \cite{CRT,DOnoho06CS} appeared. While current understanding of $\ell_1$ optimization-linear systems connection is fairly known, much less so is the case with a general $\ell_q,0<q<1$, optimization. In our recent work \cite{StojnicLqThrBnds10} we provided a study in this direction. As a result we were able to obtain a collection of lower bounds on various $\ell_q,0\leq q\leq 1$, optimization thresholds. In this paper, we provide a substantial conceptual improvement of the methodology presented in \cite{StojnicLqThrBnds10}. Moreover, the practical results in terms of achievable thresholds are also encouraging. As is usually the case with these and similar problems, the methodology we developed emphasizes their a combinatorial nature and attempts to somehow handle it. Although our results' main contributions should be on a conceptual level, they already give a very strong suggestion that $\ell_q$ optimization can in fact provide a better performance than $\ell_1$, a fact long believed to be true due to a tighter optimization relaxation it provides to the original $\ell_0$ sparsity finding oriented original problem formulation. As such, they in a way give a solid boost to further exploration of the design of the algorithms that would be able to handle $\ell_q,0<q<1$, optimization in a reasonable (if not polynomial) time.

cs.IT↗

arXiv2013

Asymmetric Little model and its ground state energies

In this paper we look at a class of random optimization problems that arise in the forms typically known in statistical physics as Little models. In \cite{BruParRit92} the Little models were studied by means of the well known tool from the statistical physics called the replica theory. A careful consideration produced a physically sound conclusion that the behavior of almost all important features of the Little models essentially resembles the behavior of the corresponding ones of appropriately scaled Sherrington-Kirkpatrick (SK) model. In this paper we revisit the Little models and consider their ground state energies as one of their main features. We then rigorously show that they indeed can be lower-bounded by the corresponding ones related to the SK model. We also provide a mathematically rigorous way to show that the replica symmetric estimate of the ground state energy is in fact a rigorous upper-bound of the ground state energy. Moreover, we then recall on a set of powerful mechanisms we recently designed for a study of the Hopfield models in \cite{StojnicHopBnds10,StojnicMoreSophHopBnds10} and show how one can utilize them to substantially lower the upper-bound that the replica symmetry theory provides.

math.OC↗

arXiv2013

Another look at the Gardner problem

In this paper we revisit one of the classical perceptron problems from the neural networks and statistical physics. In \cite{Gar88} Gardner presented a neat statistical physics type of approach for analyzing what is now typically referred to as the Gardner problem. The problem amounts to discovering a statistical behavior of a spherical perceptron. Among various quantities \cite{Gar88} determined the so-called storage capacity of the corresponding neural network and analyzed its deviations as various perceptron parameters change. In a more recent work \cite{SchTir02,SchTir03} many of the findings of \cite{Gar88} (obtained on the grounds of the statistical mechanics replica approach) were proven to be mathematically correct. In this paper, we take another look at the Gardner problem and provide a simple alternative framework for its analysis. As a result we reprove many of now known facts and rigorously reestablish a few other results.

math.OC↗

arXiv2013

Negative spherical perceptron

In this paper we consider the classical spherical perceptron problem. This problem and its variants have been studied in a great detail in a broad literature ranging from statistical physics and neural networks to computer science and pure geometry. Among the most well known results are those created using the machinery of statistical physics in \cite{Gar88}. They typically relate to various features ranging from the storage capacity to typical overlap of the optimal configurations and the number of incorrectly stored patterns. In \cite{SchTir02,SchTir03,TalBook} many of the predictions of the statistical mechanics were rigorously shown to be correct. In our own work \cite{StojnicGardGen13} we then presented an alternative way that can be used to study the spherical perceptrons as well. Among other things we reaffirmed many of the results obtained in \cite{SchTir02,SchTir03,TalBook} and thereby confirmed many of the predictions established by the statistical mechanics. Those mostly relate to spherical perceptrons with positive thresholds (which we will typically refer to as the positive spherical perceptrons). In this paper we go a step further and attack the negative counterpart, i.e. the perceptron with negative thresholds. We present a mechanism that can be used to analyze many features of such a model. As a concrete example, we specialize our results for a particular feature, namely the storage capacity. The results we obtain for the storage capacity seem to indicate that the negative case could be more combinatorial in nature and as such a somewhat harder challenge than the positive counterpart.

math.PR↗

arXiv2013

Discrete perceptrons

Perceptrons have been known for a long time as a promising tool within the neural networks theory. The analytical treatment for a special class of perceptrons started in seminal work of Gardner \cite{Gar88}. Techniques initially employed to characterize perceptrons relied on a statistical mechanics approach. Many of such predictions obtained in \cite{Gar88} (and in a follow-up \cite{GarDer88}) were later on established rigorously as mathematical facts (see, e.g. \cite{SchTir02,SchTir03,TalBook,StojnicGardGen13,StojnicGardSphNeg13,StojnicGardSphErr13}). These typically related to spherical perceptrons. A lot of work has been done related to various other types of perceptrons. Among the most challenging ones are what we will refer to as the discrete perceptrons. An introductory statistical mechanics treatment of such perceptrons was given in \cite{GutSte90}. Relying on results of \cite{Gar88}, \cite{GutSte90} characterized many of the features of several types of discrete perceptrons. We in this paper, consider a similar subclass of discrete perceptrons and provide a mathematically rigorous set of results related to their performance. As it will turn out, many of the statistical mechanics predictions obtained for discrete predictions will in fact appear as mathematically provable bounds. This will in a way emulate a similar type of behavior we observed in \cite{StojnicGardGen13,StojnicGardSphNeg13,StojnicGardSphErr13} when studying spherical perceptrons.

math.PR↗

arXiv2013

A rigorous geometry-probability equivalence in characterization of $\ell_1$-optimization

In this paper we consider under-determined systems of linear equations that have sparse solutions. This subject attracted enormous amount of interest in recent years primarily due to influential works \cite{CRT,DonohoPol}. In a statistical context it was rigorously established for the first time in \cite{CRT,DonohoPol} that if the number of equations is smaller than but still linearly proportional to the number of unknowns then a sparse vector of sparsity also linearly proportional to the number of unknowns can be recovered through a polynomial $\ell_1$-optimization algorithm (of course, this assuming that such a sparse solution vector exists). Moreover, the geometric approach of \cite{DonohoPol} produced the exact values for the proportionalities in question. In our recent work \cite{StojnicCSetam09} we introduced an alternative statistical approach that produced attainable values of the proportionalities. Those happened to be in an excellent numerical agreement with the ones of \cite{DonohoPol}. In this paper we give a rigorous analytical confirmation that the results of \cite{StojnicCSetam09} indeed match those from \cite{DonohoPol}.

cs.IT↗

arXiv2013

Upper-bounding $\ell_1$-optimization weak thresholds

In our recent work \cite{StojnicCSetam09} we considered solving under-determined systems of linear equations with sparse solutions. In a large dimensional and statistical context we proved that if the number of equations in the system is proportional to the length of the unknown vector then there is a sparsity (number of non-zero elements of the unknown vector) also proportional to the length of the unknown vector such that a polynomial $\ell_1$-optimization technique succeeds in solving the system. We provided lower bounds on the proportionality constants that are in a solid numerical agreement with what one can observe through numerical experiments. Here we create a mechanism that can be used to derive the upper bounds on the proportionality constants. Moreover, the upper bounds obtained through such a mechanism match the lower bounds from \cite{StojnicCSetam09} and ultimately make the latter ones optimal.

cs.IT↗

arXiv2013

A framework to characterize performance of LASSO algorithms

In this paper we consider solving \emph{noisy} under-determined systems of linear equations with sparse solutions. A noiseless equivalent attracted enormous attention in recent years, above all, due to work of \cite{CRT,CanRomTao06,DonohoPol} where it was shown in a statistical and large dimensional context that a sparse unknown vector (of sparsity proportional to the length of the vector) can be recovered from an under-determined system via a simple polynomial $\ell_1$-optimization algorithm. \cite{CanRomTao06} further established that even when the equations are \emph{noisy}, one can, through an SOCP noisy equivalent of $\ell_1$, obtain an approximate solution that is (in an $\ell_2$-norm sense) no further than a constant times the noise from the sparse unknown vector. In our recent works \cite{StojnicCSetam09,StojnicUpper10}, we created a powerful mechanism that helped us characterize exactly the performance of $\ell_1$ optimization in the noiseless case (as shown in \cite{StojnicEquiv10} and as it must be if the axioms of mathematics are well set, the results of \cite{StojnicCSetam09,StojnicUpper10} are in an absolute agreement with the corresponding exact ones from \cite{DonohoPol}). In this paper we design a mechanism, as powerful as those from \cite{StojnicCSetam09,StojnicUpper10}, that can handle the analysis of a LASSO type of algorithm (and many others) that can be (or typically are) used for "solving" noisy under-determined systems. Using the mechanism we then, in a statistical context, compute the exact worst-case $\ell_2$ norm distance between the unknown sparse vector and the approximate one obtained through such a LASSO. The obtained results match the corresponding exact ones obtained in \cite{BayMon10,DonMalMon10}. Moreover, as a by-product of our analysis framework we recognize existence of an SOCP type of algorithm that achieves the same performance.

cs.IT↗

arXiv2013

Regularly random duality

In this paper we look at a class of random optimization problems. We discuss ways that can help determine typical behavior of their solutions. When the dimensions of the optimization problems are large such an information often can be obtained without actually solving the original problems. Moreover, we also discover that fairly often one can actually determine many quantities of interest (such as, for example, the typical optimal values of the objective functions) completely analytically. We present a few general ideas and emphasize that the range of applications is enormous.

cs.IT↗