arXiv subjects

Mihailo Stojnic

Publications and source records attributed to Mihailo Stojnic.

At least 55 records · Page 3Linked to original sources

arXiv2023

Causal Inference (C-inf) -- closed form worst case typical phase transitions

In this paper we establish a mathematically rigorous connection between Causal inference (C-inf) and the low-rank recovery (LRR). Using Random Duality Theory (RDT) concepts developed in [46,48,50] and novel mathematical strategies related to free probability theory, we obtain the exact explicit typical (and achievable) worst case phase transitions (PT). These PT precisely separate scenarios where causal inference via LRR is possible from those where it is not. We supplement our mathematical analysis with numerical experiments that confirm the theoretical predictions of PT phenomena, and further show that the two closely match for fairly small sample sizes. We obtain simple closed form representations for the resulting PTs, which highlight direct relations between the low rankness of the target C-inf matrix and the time of the treatment. Hence, our results can be used to determine the range of C-inf's typical applicability.

stat.ML↗

arXiv2023

Causal Inference (C-inf) -- asymmetric scenario of typical phase transitions

In this paper, we revisit and further explore a mathematically rigorous connection between Causal inference (C-inf) and the Low-rank recovery (LRR) established in [10]. Leveraging the Random duality - Free probability theory (RDT-FPT) connection, we obtain the exact explicit typical C-inf asymmetric phase transitions (PT). We uncover a doubling low-rankness phenomenon, which means that exactly two times larger low rankness is allowed in asymmetric scenarios compared to the symmetric worst case ones considered in [10]. Consequently, the final PT mathematical expressions are as elegant as those obtained in [10], and highlight direct relations between the targeted C-inf matrix low rankness and the time of treatment. Our results have strong implications for applications, where C-inf matrices are not necessarily symmetric.

stat.ML↗

arXiv2020

Algorithmic random duality theory -- large scale CLuP

Based on our \bl{\textbf{Random Duality Theory (RDT)}}, in a sequence of our recent papers \cite{Stojnicclupint19,Stojnicclupcmpl19,Stojnicclupplt19}, we introduced a powerful algorithmic mechanism (called \bl{\textbf{CLuP}}) that can be utilized to solve \textbf{\emph{exactly}} NP hard optimization problems in polynomial time. Here we move things further and utilize another of remarkable RDT features that we established in a long line of work in \cite{StojnicCSetam09,StojnicCSetamBlock09,StojnicISIT2010binary,StojnicDiscPercp13,StojnicUpper10,StojnicGenLasso10,StojnicGenSocp10,StojnicPrDepSocp10,StojnicRegRndDlt10,Stojnicbinary16fin,Stojnicbinary16asym}. Namely, besides being stunningly precise in characterizing the performance of various random structures and optimization problems, RDT simultaneously also provided an almost unparallel way for creating computationally efficient optimization algorithms that achieve such performance. One of the keys to our success was our ability to transform the initial \textbf{\emph{constrained}} optimization into an \textbf{\emph{unconstrained}} one and in doing so greatly simplify things both conceptually and computationally. That ultimately enabled us to solve a large set of classical optimization problems on a very large scale level. Here, we demonstrate how such a thinking can be applied to CLuP as well and eventually utilized to solve pretty much any problem that the basic CLuP from \cite{Stojnicclupint19,Stojnicclupcmpl19,Stojnicclupplt19} can solve.

cs.IT↗

arXiv2020

Rephased CLuP

In \cite{Stojnicclupint19,Stojnicclupcmpl19,Stojnicclupplt19} we introduced CLuP, a \bl{\textbf{Random Duality Theory (RDT)}} based algorithmic mechanism that can be used for solving hard optimization problems. Due to their introductory nature, \cite{Stojnicclupint19,Stojnicclupcmpl19,Stojnicclupplt19} discuss the most fundamental CLuP concepts. On the other hand, in our companion paper \cite{Stojniccluplargesc20} we started the story of going into a bit deeper details that relate to many of other remarkable CLuP properties with some of them reaching well beyond the basic fundamentals. Namely, \cite{Stojniccluplargesc20} discusses how a somewhat silent RDT feature (its algorithmic power) can be utilized to ensure that CLuP can be run on very large problem instances as well. In particular, applying CLuP to the famous MIMO ML detection problem we showed in \cite{Stojniccluplargesc20} that its a large scale variant, $\text{CLuP}^{r_0}$, can handle with ease problems with \textbf{\emph{several thousands}} of unknowns with theoretically minimal complexity per iteration (only a single matrix-vector multiplication suffices). In this paper we revisit MIMO ML detection and discuss another remarkable phenomenon that emerges within the CLuP structure, namely the so-called \bl{\textbf{\emph{rephasing}}}. As MIMO ML enters the so-called low $α$ regime (fat system matrix with ratio of the number of rows and columns, $α$, going well below $1$) it becomes increasingly difficult even for the basic standard CLuP to handle it. However, the discovery of the rephasing ensures that CLuP remains on track and preserves its ability to achieve the ML performance. To demonstrate the power of the rephasing we also conducted quite a few numerical experiments, compared the results we obtained through them to the theoretical predictions, and observed an excellent agreement.

cs.IT↗

arXiv2020

Sparse linear regression -- CLuP achieves the ideal \emph{exact} ML

In this paper we revisit one of the classical statistical problems, the so-called sparse maximum-likelihood (ML) linear regression. As a way of attacking this type of regression, we present a novel CLuP mechanism that to a degree relies on the \bl{\textbf{Random Duality Theory (RDT)}} based algorithmic machinery that we recently introduced in \cite{Stojnicclupint19,Stojnicclupcmpl19,Stojnicclupplt19,Stojniccluplargesc20,Stojniccluprephased20}. After the initial success that the CLuP exhibited in achieving the exact ML performance while maintaining excellent computational complexity related properties in MIMO ML detection in \cite{Stojnicclupint19,Stojnicclupcmpl19,Stojnicclupplt19}, one would naturally expect that a similar type of success can be achieved in other ML considerations. The results that we present here confirm that such an expectation is indeed reasonable. In particular, within the sparse regression context, the introduced CLuP mechanism indeed turns out to be able to \bl{\textbf{\emph{achieve the ideal ML performance}}}. Moreover, it can substantially outperform some of the most prominent earlier state of the art algorithmic concepts, among them even the variants of the famous LASSO and SOCP from \cite{StojnicPrDepSocp10,StojnicGenLasso10,StojnicGenSocp10}. Also, our recent results presented in \cite{Stojniccluplargesc20,Stojniccluprephased20} showed that the CLuP has excellent \bl{\textbf{\emph{large-scale}}} and the so-called \bl{\textbf{\emph{rephasing}}} abilities. Since such large-scale algorithmic features are possibly even more desirable within the sparse regression context we here also demonstrate that the basic CLuP ideas can be reformulated to enable solving with a relative ease the regression problems with \bl{\textbf{\emph{several thousands}}} of unknowns.

cs.IT↗

arXiv2019

Complexity analysis of the Controlled Loosening-up (CLuP) algorithm

In our companion paper \cite{Stojnicclupint19} we introduced a powerful mechanism that we referred to as the Controlled Loosening-up (CLuP) for handling MIMO ML-detection problems. It turned out that the algorithm has many remarkable features and one of them, the \emph{computational complexity}, we discuss in more details in this paper. As was explained in \cite{Stojnicclupint19}, the CLuP is an iterative procedure where each iteration amounts to solving a simple quadratic program. This clearly implies that the key contributing factor to its overall computational complexity is the number of iterations needed to achieve a required precision. As was also hinted in \cite{Stojnicclupint19}, that number seems to be fairly low and in some of the most interesting scenarios often not even larger than $10$. Here we provide a Random Duality Theory based careful analysis that indeed indicates that a very small number of iterations is sufficient to achieve an excellent performance. A solid set of results obtained through numerical experiments is presented as well and shown to be in a nice agreement with what the theoretical analysis predicts. Also, as was the case in \cite{Stojnicclupint19}, we again focus only on the core CLuP algorithm but do mention on several occasions that the concepts that we introduce here are as remarkably general as those that we introduced in \cite{Stojnicclupint19} and can be utilized in the analysis of a large number of classes of algorithms applicable in the most diverse of scientific fields. Many results in these directions we will present in several of our companion papers.

cs.IT↗

arXiv2019

Controlled Loosening-up (CLuP) -- achieving exact MIMO ML in polynomial time

In this paper we attack one of the most fundamental signal processing/informaton theory problems, widely known as the MIMO ML-detection. We introduce a powerful Random Duality Theory (RDT) mechanism that we refer to as the Controlled Loosening-up (CLuP) as a way of achieving the exact ML-performance in MIMO systems in polynomial time. We first outline the general strategy and then discuss the rationale behind the entire concept. A solid collection of results obtained through numerical experiments is presented as well and found to be in an excellent agreement with what the theory predicts. As this is the introductory paper of a massively general concept that we have developed, we mainly focus on keeping things as simple as possible and put the emphasis on the most fundamental ideas. In our several companion papers we present various other complementary results that relate to both, theoretical and practical aspects and their connections to a large collection of other problems and results that we have achieved over the years in Random Duality.

cs.IT↗

arXiv2019

Starting CLuP with polytope relaxation

The Controlled Loosening-up (CLuP) mechanism that we recently introduced in \cite{Stojnicclupint19} is a generic concept that can be utilized to solve a large class of problems in polynomial time. Since it relies in its core on an iterative procedure, the key to its excellent performance lies in a typically very small number of iterations needed to execute the entire algorithm. In a separate paper \cite{Stojnicclupcmpl19}, we presented a detailed complexity analysis that indeed confirms the relatively small number of iterations. Since both papers, \cite{Stojnicclupint19} and \cite{Stojnicclupcmpl19} are the introductory papers on the topic we made sure to limit the initial discussion just to the core of the algorithm and consequently focused only on the algorithm's most basic version. On numerous occasions though, we emphasized that various improvements and further upgrades are possible. In this paper we present a first step in this direction and discuss a very simple upgrade that can be introduced on top of the basic CLuP mechanism. It relates to the starting of the CLuP and suggests the well-known so-called polytope-relaxation heuristic (see, e.g. \cite{StojnicBBSD05,StojnicBBSD08}) as the starting point. We refer to this variant of CLuP as the CLuP-plt and proceed with the presentation of its complexity analysis. As in \cite{Stojnicclupcmpl19}, a particular \textbf{\emph{complexity analysis per iteration level}} type of complexity analysis is chosen and presented through the algorithm's application on the well-known MIMO ML detection problem. As expected, the analysis confirms that CLuP-plt performs even better than the original CLuP. In some of the most interesting regimes it often achieves within the \textbf{\emph{first three iterations}} an excellent performance. We also complement the theoretical findings with a solid set of numerical experiments.

cs.IT↗

arXiv2016

Generic and lifted probabilistic comparisons -- max replaces minmax

In this paper we introduce a collection of powerful statistical comparison results. We first present the results that we obtained while developing a general comparison concept. After that we introduce a separate lifting procedure that is a comparison concept on its own. We then show how in certain scenarios the lifting procedure basically represents a substantial upgrade over the general strategy. We complement the introduced results with a fairly large collection of numerical experiments that are in an overwhelming agreement with what the theory predicts. We also show how many well known comparison results (e.g. Slepian's max and Gordon's minmax principle) can be obtained as special cases. Moreover, it turns out that the minmax principle can be viewed as a single max principle as well. The range of applications is enormous. It starts with revisiting many of the results we created in recent years in various mathematical fields and recognizing that they are fully self-contained as their starting blocks are specialized variants of the concepts introduced here. Further upgrades relate to core comparison extensions on the one side and more practically oriented modifications on the other. Those that we deem the most important we discuss in several separate companion papers to ensure preserving the introductory elegance and simplicity of what is presented here.

math.PR↗

arXiv2016

Fully bilinear generic and lifted random processes comparisons

In our companion paper \cite{Stojnicgscomp16} we introduce a collection of fairly powerful statistical comparison results. They relate to a general comparison concept and its an upgrade that we call lifting procedure. Here we provide a different generic principle (which we call fully bilinear) that in certain cases turns out to be stronger than the corresponding one from \cite{Stojnicgscomp16}. Moreover, we also show how the principle that we introduce here can also be pushed through the lifting machinery of \cite{Stojnicgscomp16}. Finally, as was the case in \cite{Stojnicgscomp16}, here we also show how the well known Slepian's max and Gordon's minmax comparison principles can be obtained as special cases of the mechanisms that we present here. We also create their lifted upgrades which happen to be stronger than the corresponding ones in \cite{Stojnicgscomp16}. A fairly large collection of results obtained through numerical experiments is also provided. It is observed that these results are in an excellent agreement with what the theory predicts.

math.PR↗

arXiv2016

Partial $\ell_1$ optimization in random linear systems -- phase transitions and large deviations

$\ell_1$ optimization is a well known heuristic often employed for solving various forms of sparse linear problems. In this paper we look at its a variant that we refer to as the \emph{partial} $\ell_1$ and discuss its mathematical properties when used for solving linear under-determined systems of equations. We will focus on large random systems and discuss the phase transition (PT) phenomena and how they connect to the large deviation principles (LDP). Using a variety of probabilistic and geometric techniques that we have developed in recent years we will first present general guidelines that conceptually fully characterize both, the PTs and the LDPs. After that we will put an emphasis on providing a collection of explicit analytical solutions to all of the underlying mathematical problems. As a nice bonus to the developed concepts, the forms of the analytical solutions will, in our view, turn out to be fairly elegant as well.

math.OC↗

arXiv2016

Partial $\ell_1$ optimization in random linear systems -- finite dimensions

In this paper we provide a complementary set of results to those we present in our companion work \cite{Stojnicl1HidParasymldp} regarding the behavior of the so-called partial $\ell_1$ (a variant of the standard $\ell_1$ heuristic often employed for solving under-determined systems of linear equations). As is well known through our earlier works \cite{StojnicICASSP10knownsupp,StojnicTowBettCompSens13}, the partial $\ell_1$ also exhibits the phase-transition (PT) phenomenon, discovered and well understood in the context of the standard $\ell_1$ through Donoho's and our own works \cite{DonohoPol,DonohoUnsigned,StojnicCSetam09,StojnicUpper10}. \cite{Stojnicl1HidParasymldp} goes much further though and, in addition to the determination of the partial $\ell_1$'s phase-transition curves (PT curves) (which had already been done in \cite{StojnicICASSP10knownsupp,StojnicTowBettCompSens13}), provides a substantially deeper understanding of the PT phenomena through a study of the underlying large deviations principles (LDPs). As the PT and LDP phenomena are by their definitions related to large dimensional settings, both sets of our works, \cite{StojnicICASSP10knownsupp,StojnicTowBettCompSens13} and \cite{Stojnicl1HidParasymldp}, consider what is typically called the asymptotic regime. In this paper we move things in a different direction and consider finite dimensional scenarios. Basically, we provide explicit performance characterizations for any given collection of systems/parameters dimensions. We do so for two different variants of the partial $\ell_1$, one that we call exactly the partial $\ell_1$ and another one, possibly a bit more practical, that we call the hidden partial $\ell_1$.

math.OC↗

arXiv2016

Random linear under-determined systems with block-sparse solutions -- asymptotics, large deviations, and finite dimensions

In this paper we consider random linear under-determined systems with block-sparse solutions. A standard subvariant of such systems, namely, precisely the same type of systems without additional block structuring requirement, gained a lot of popularity over the last decade. This is of course in first place due to the success in mathematical characterization of an $\ell_1$ optimization technique typically used for solving such systems, initially achieved in \cite{CRT,DOnoho06CS} and later on perfected in \cite{DonohoPol,DonohoUnsigned,StojnicCSetam09,StojnicUpper10}. The success that we achieved in \cite{StojnicCSetam09,StojnicUpper10} characterizing the standard sparse solutions systems, we were then able to replicate in a sequence of papers \cite{StojnicCSetamBlock09,StojnicUpperBlock10,StojnicICASSP09block,StojnicJSTSP09} where instead of the standard $\ell_1$ optimization we utilized its an $\ell_2/\ell_1$ variant as a better fit for systems with block-sparse solutions. All of these results finally settled the so-called threshold/phase transitions phenomena (which naturally assume the asymptotic/large dimensional scenario). Here, in addition to a few novel asymptotic considerations, we also try to raise the level a bit, step a bit away from the asymptotics, and consider the finite dimensions scenarios as well.

math.OC↗

arXiv2016

Box constrained $\ell_1$ optimization in random linear systems -- asymptotics

In this paper we consider box constrained adaptations of $\ell_1$ optimization heuristic when applied for solving random linear systems. These are typically employed when on top of being sparse the systems' solutions are also known to be confined in a specific way to an interval on the real axis. Two particular $\ell_1$ adaptations (to which we will refer as the \emph{binary} $\ell_1$ and \emph{box} $\ell_1$) will be discussed in great detail. Many of their properties will be addressed with a special emphasis on the so-called phase transitions (PT) phenomena and the large deviation principles (LDP). We will fully characterize these through two different mathematical approaches, the first one that is purely probabilistic in nature and the second one that connects to high-dimensional geometry. Of particular interest we will find that for many fairly hard mathematical problems a collection of pretty elegant characterizations of their final solutions will turn out to exist.

math.PR↗

arXiv2016

Box constrained $\ell_1$ optimization in random linear systems -- finite dimensions

Our companion work \cite{Stojnicl1BnBxasymldp} considers random under-determined linear systems with box-constrained sparse solutions and provides an asymptotic analysis of a couple of modified $\ell_1$ heuristics adjusted to handle such systems (we refer to these modifications of the standard $\ell_1$ as binary and box $\ell_1$). Our earlier work \cite{StojnicISIT2010binary} established that the binary $\ell_1$ does exhibit the so-called phase-transition phenomenon (basically the same phenomenon well-known through earlier considerations to be a key feature of the standard $\ell_1$, see, e.g. \cite{DonohoPol,DonohoUnsigned,StojnicCSetam09,StojnicUpper10}). Moreover, in \cite{StojnicISIT2010binary}, we determined the precise location of the co-called phase-transition (PT) curve. On the other hand, in \cite{Stojnicl1BnBxasymldp} we provide a much deeper understanding of the PTs and do so through a large deviations principles (LDP) type of analysis. In this paper we complement the results of \cite{Stojnicl1BnBxasymldp} by leaving the asymptotic regime naturally assumed in the PT and LDP considerations aside and instead working in a finite dimensional setting. Along the same lines, we provide for both, the binary and the box $\ell_1$, precise finite dimensional analyses and essentially determine their ultimate statistical performance characterizations. On top of that, we explain how the results created here can be utilized in the asymptotic setting, considered in \cite{Stojnicl1BnBxasymldp}, as well. Finally, for the completeness, we also present a collection of results obtained through numerical simulations and observe that they are in a massive agreement with our theoretical calculations.

math.OC↗

arXiv2016

Random linear systems with sparse solutions -- finite dimensions

In our companion work \cite{Stojnicl1RegPosasymldp} we revisited random under-determined linear systems with sparse solutions. The main emphasis was on the performance analysis of the $\ell_1$ heuristic in the so-called asymptotic regime, i.e. in the regime where the systems' dimensions are large. Through an earlier sequence of work \cite{DonohoPol,DonohoUnsigned,StojnicCSetam09,StojnicUpper10}, it is now well known that in such a regime the $\ell_1$ exhibits the so-called \emph{phase transition} (PT) phenomenon. \cite{Stojnicl1RegPosasymldp} then went much further and established the so-called \emph{large deviations principle} (LDP) type of behavior that characterizes not only the breaking points of the $\ell_1$'s success but also the behavior in the entire so-called \emph{transition zone} around these points. Both of these concepts, the PTs and the LDPs, are in fact defined so that one can use them to characterize the asymptotic behavior. In this paper we complement the results of \cite{Stojnicl1RegPosasymldp} by providing an exact detailed analysis in the non-asymptotic regime. Of course, not only are the non-asymptotic results complementing those from \cite{Stojnicl1RegPosasymldp}, they actually are the ones that ultimately fully characterize the $\ell_1$'s behavior in the most general sense. We introduce several novel high-dimensional geometry type of strategies that enable us to eventually determine the $\ell_1$'s behavior.

math.OC↗

arXiv2016

Random linear systems with sparse solutions -- asymptotics and large deviations

In this paper we revisit random linear under-determined systems with sparse solutions. We consider $\ell_1$ optimization heuristic known to work very well when used to solve these systems. A collection of fundamental results that relate to its performance analysis in a statistical scenario is presented. We start things off by recalling on now classical phase transition (PT) results that we derived in \cite{StojnicCSetam09,StojnicUpper10}. As these represent the so-called breaking point characterizations, we now complement them by analyzing the behavior in a zone around the breaking points in a sense typically used in the study of the large deviation properties (LDP) in the classical probability theory. After providing a conceptual solution to these problems we attack them on a "hardcore" mathematical level attempting/hoping to be able to obtain explicit solutions as elegant as those we obtained in \cite{StojnicCSetam09,StojnicUpper10} (this time around though, the final characterizations were to be expected to be way more involved than in \cite{StojnicCSetam09,StojnicUpper10}, simply, the ultimate goals are set much higher and their achieving would provide a much richer collection of information about the $\ell_1$'s behavior). Perhaps surprisingly, the final LDP $\ell_1$ characterizations that we obtain happen to match the elegance of the corresponding PT ones from \cite{StojnicCSetam09,StojnicUpper10}. Moreover, as we have done in \cite{StojnicEquiv10}, here we also present a corresponding LDP set of results that can be obtained through an alternative high-dimensional geometry approach. Finally, we also prove that the two types of characterizations, obtained through two substantially different mathematical approaches, match as one would hope that they do.

math.OC↗

arXiv2015

Linear under-determined systems with sparse solutions: Redirecting a challenge?

Seminal works \cite{CRT,DonohoUnsigned,DonohoPol} generated a massive interest in studying linear under-determined systems with sparse solutions. In this paper we give a short mathematical overview of what was accomplished in last 10 years in a particular direction of such a studying. We then discuss what we consider were the main challenges in last 10 years and give our own view as to what are the main challenges that lie ahead. Through the presentation we arrive to a point where the following natural rhetoric question arises: is it a time to redirect the main challenges? While we can not provide the answer to such a question we hope that our small discussion will stimulate further considerations in this direction.

cs.IT↗