# A Spin Glass Model for Reconstructing Nonlinearly Encrypted Signals Corrupted by Noise

- 38 Downloads

## Abstract

We define a (symmetric key) encryption of a signal \(\mathbf{s}\in {\mathbb {R^N}}\) as a random mapping \(\mathbf{s}\mapsto \mathbf y =(y_1,\ldots ,y_M)^T\in \mathbb {R}^M\) known both to the sender and a recipient. In general the recipients may have access only to images \(\mathbf{y}\) corrupted by an additive noise of unknown strength. Given the encryption redundancy parameter (ERP) \(\mu =M/N\ge 1\) and the signal strength parameter \(R=\sqrt{\sum _i {s_i^2/N}}\), we consider the problem of reconstructing \(\mathbf{s}\) from its corrupted image by a Least Square Scheme for a certain class of random Gaussian mappings. The problem is equivalent to finding the configuration of minimal energy in a certain version of spherical spin glass model, with *squared* Gaussian random interaction potential. We use the Parisi replica symmetry breaking scheme to evaluate the mean overlap \(p_{\infty }\in [0,1]\) between the original signal and its recovered image (known as ’estimate’) as \(N\rightarrow \infty \), for a given (’bare’) noise-to-signal ratio (NSR) \(\gamma \ge 0\). Such an overlap is a measure of the quality of the signal reconstruction. We explicitly analyze the general case of *linear-quadratic* family of random mappings and discuss the full \(p_{\infty } (\gamma )\) curve. When nonlinearity exceeds a certain threshold but redundancy is not yet too big, the replica symmetric solution is necessarily broken in some interval of NSR. We show that encryptions with a nonvanishing linear component permit reconstructions with \(p_{\infty }>0\) for any \(\mu >1\) and any \(\gamma <\infty \), with \(p_{\infty }\sim \gamma ^{-1/2}\) as \(\gamma \rightarrow \infty \). In contrast, for the case of *purely quadratic* nonlinearity, for any ERP \(\mu >1\) there exists a threshold NSR value \(\gamma _c(\mu )\) such that \(p_{\infty }=0\) for \(\gamma >\gamma _c(\mu )\) making the reconstruction impossible. The behaviour close to the threshold is given by \(p_{\infty }\sim (\gamma _c-\gamma )^{3/4}\) and is controlled by the replica symmetry breaking mechanism.

## Keywords

Spin glass Signal reconstruction Inference## 1 Introduction

### 1.1 Description of the Problem

*R*via the Euclidean norm as \(R=\sqrt{\frac{1}{N}\left( \mathbf{s},\mathbf{s}\right) }\), where \((\cdot ,\cdot )\) stands for the Euclidean inner product in \(\mathbb {R}^N\). By a (symmetric key)

*encryption*of the source signal we understand a

*random mapping*\(\mathbf{s}\mapsto \mathbf y ={\left( \begin{array}{c}y_1 \\ \ldots \\ y_M\end{array}\right) }\in \mathbb {R}^M\) known both to the sender and a recipient. For further reference we find it useful to write the mapping component-wise explicitly as

*fixed*vector, and then employ the Least-Square reconstruction scheme, which for a given set of observations \(z_k=V_k(\mathbf{s})+b_k\) returns an estimate of the input signal as

To avoid misunderstanding, we briefly contrast our approach with the Bayesian inference philosophy. In the latter framework one assumes that the probability density \(\rho (\mathbf{w})\) over the set \(\mathbb {W}\) of feasible input signals (the ’prior’ distribution) is either known or one can make an Ansatz on its form. The Bayes theorem provides the *a posteori* probability density \(p(\mathbf{w}|\mathbf{z})\) of the input signal \(\mathbf{w}\) for a given observation \(\mathbf{z}\) in the form \(p(\mathbf{w}|\mathbf{z})=p(\mathbf{z}|\mathbf{w})\rho (\mathbf{w})/\int _{\mathbb {W}} p(\mathbf{z}|\mathbf{w})\,\rho (\mathbf{w})\,d\mathbf{w}\), where \(p(\mathbf{z}|\mathbf{w})\) is the probability to observe \(\mathbf{z}\) for a given input \(\mathbf{w}\). In Bayesian inference one then seeks to minimize some *expected* (with respect to \(p(\mathbf{w}|\mathbf{z})\)) error/loss function. For example, if one seeks to minimize the *mean square error* loss function (MMSE), the optimal estimate is given by the mean \(\mathbf{x}_{MMSE}:=\int _{\mathbb {W}} \mathbf{w} p(\mathbf{w}|\mathbf{z})\,d\mathbf{w}\). In our model with the Gaussian noise \(\mathbf{b}\sim \mathcal{N}(\mathbf{0},\sigma ^2 \mathbf{1}_M)\) the probability \(p(\mathbf{z}|\mathbf{w})\) to observe \(\mathbf{z}\) is given by \(p(\mathbf{z}|\mathbf{w})\sim \exp \left\{ -\sum _{k=1}^M \frac{ \left( z_k-V_k(\mathbf {w})\right) ^2}{2\sigma ^2}\right\} \). Therefore evaluation of \(\mathbf{x}_{MMSE}\) requires that the noise strength \(\sigma ^2\) is explicitly known to the recipient. Such a knowledge is formally not required in the estimate Eq. (2) used in the present paper. Instead we note that our approach can be given a formal Bayesian meaning as a Maximum–A-Posteriori (MAP) estimator (optimal under ’hit-or-miss’ or \(0-1\) loss function) with a uniform prior distribution \(\rho (\mathbf{w})=const\) over the feasibility set \(\mathbb {W}\). In such a case it also incidentally coincides with the maximal likelihood (ML) estimator which seeks to maximize the probability \(p(\mathbf{z}|\mathbf{w})\) over \(\mathbf{w}\) for the observation \(\mathbf{z}\) corresponding to the actual input signal \(\mathbf{s}\).

*M*remains smaller than

*N*, any solution \(\mathbf{x}\in \mathbb {W}\subseteq \mathbb {R}^N\) of the set of

*M*equations \(b_k+V_k(\mathbf{s})-V_k(\mathbf{x})=0,\,\, k=1,\ldots , M\) will be corresponding to the exactly zero value of the cost function, and could be used as a legitimate estimate. The set of estimates then form continuously parametrized manifolds in \(\mathbb {R}^N\). It is therefore clear that even in the absence of any noise full reconstruction of the signal for \(M<N\) under this scheme is impossible. Although such a case is not at all devoid of interest, we do not treat it in the present paper leaving it for a separate study. In contrast, for ’redundantantly’ encrypted signals with \(M\ge N\) the set of possible estimates generically consists of isolated points in \(\mathbb {R}^N\). To this end we introduce the Encryption Redundancy Parameter (ERP) \(\mu =M/N\in (1,\infty )\). We will see that under such conditions signals can be in general faithfully reconstructed in some range of the noise-to-signal ratios \(\gamma >0\).

In this paper we are going to apply tools of Statistical Mechanics for calculating the average asymptotic distortion for a certain class of the least square reconstruction of a randomly encrypted noisy signal. As this is essentially a large-scale random optimization problem, methods of statistical mechanics of disordered systems like the replica trick developed in theory of spin glasses are known to be efficient in providing important analytical insights in the statistical properties of the solution, see e.g. [31, 34]. It is also worth noting that distinctly different aspects of the problem of information reconstruction ( the so-called *error-correcting* procedures) were discussed in the framework of spin glass ideas already in the seminal work by Sourlas [39].

*R*, and therefore can restrict the least square minimization search in Eq. (2) to the feasibility set \(\mathbb {W}\) given by \((N-1)-\)dimensional sphere of the radius \(R\sqrt{N}\). We will refer to such a condition as the ’spherical constraint’. From the point of view of the Bayesian analysis our reconstruction scheme can be considered as a MAP estimator with postulated prior distribution being the uniform measure on the above-mentioned \((N-1)-\)dimensional sphere. As the lengths of both the input signal \(\mathbf{s}\) and an estimator \(\mathbf{x}\) are fixed to \(R\sqrt{N}\), the distance Eq. (3) depends only on the scalar product \((\mathbf{x},\mathbf{s})\). We therefore can conveniently characterize the quality of the reconstruction via the

*quality parameter*defined as

*R*is technically convenient but can be further relaxed; the analysis can be extended, without much difficulty, to the search in a spherical shell \(R_1\sqrt{N}\le |\mathbf{w}| \le R_2\sqrt{N}\), and allows to include additive penalty terms which do not violate the rotational symmetry, e.g. squared Euclidean norm of the signal, and hopefully to some other situations.

*isotropic*gaussian-distributed random fields on the sphere, independent for different values of

*k*(and independent of the noise \(\mathbf{b}\)) , with mean-zero and the covariance structure dependent only on the angle between the vectors. Using the scaling appropriate for our problem in the limit of large

*N*we represent such covariances as

*linear*encryption algorithm, with the functions \(V_k(\mathbf x ), \quad k=1,\ldots , M\) chosen in the form of random linear combinations:

*N*i.i.d. components characterized by the variances \(\left\langle a_{ki}a_{lj}\right\rangle =\frac{J_1^2}{N}\delta _{lk}\delta _{ij}\). Such choice implies the covariance Eq. (5) with \(\Phi \left( u\right) =J_1^2\, u\).

*A*whose

*M*rows are represented by (transposed) vectors \(\mathbf{a}_k^T\) featuring in Eq. (6). We than can easily see that the stationarity conditions in that case amount to the following matrix equation:

*N*in \(\lambda \). The number of real solutions of that equation depends on the noise vector \(\mathbf{b}\). One of the real solutions corresponds to the minimum of the cost function, others to saddle-points or maxima. In particular, in the (trivial) limiting case of no noise \(\mathbf{b}=0\) the global minimum corresponds to \(\lambda =0\) implying reconstruction with no distortion: \(\mathbf{x}=\mathbf{s}\), hence \(p_N=1\) as is natural to expect. At the same time, for any \(\mathbf{b}\ne 0\) the analysis of Eq. (9) becomes a non-trivial problem. One possible way is to account for the presence of a weak noise with small variance \(\sigma ^2\) by developing a perturbation theory in the small scaled NSR parameter \(\tilde{\gamma }=\frac{\sigma ^2}{J_1^2R^2}\ll 1\). Such a theory is outlined in the Appendix A, where we find that for a given value of ERP \(\mu >1\) and in the leading order in \(\tilde{\gamma }\) the asymptotic disorder-averaged quality reconstruction parameter defined in Eq. (4) is given by:

Although the perturbation theories are conceptually straightforward, and can be with due effort extended to higher orders, the calculations quickly become too cumbersome. At the moment we are not aware of any direct approach to our minimization problem in the linear encryption case which may provide non-perturbative results, as \(N\rightarrow \infty \), for asymptotic distortion \(p_{\infty }\) at values of scaled NSR parameter \(\tilde{\gamma }\) of the order of unity. At the same time we will see that methods of statistical mechanics provide a very explicit expression for any \(\tilde{\gamma }\).

It is necessary to mention that various instances of not dissimilar *linear* reconstruction problems in related forms received recently a considerable attention. The emphasis in those studies seems however to be mainly restricted to the case of source signals being subject to a compressed sensing, i.e. represented by a sparse vector with a finite fraction of zero entries, see e.g. [28, 32, 42] and references therein. To this end especially deserve mentioning the works [7, 8, 9] which studied the mean value of distortions for MAP estimator for a linear problem (though with prior distribution different from the spherical constraint). More recently, a very general signal reconstruction problem for nonlinear encryption functions of a linear random mapping was considered in the frameework of statistical-mechanics based Bayesian approach in [6]. One may also mention a particular case of *bi-linear* reconstruction considered in a similar framework in [38]. Although having a moderate overlap with methods used in this work, the actual calculations and the main message of those papers seem rather different.

*nonlinear*random Gaussian encryptions

^{1}. The corresponding class of functions \(V_k(\mathbf x )\) extends the above-mentioned case of random linear forms to higher-order random forms, the first nontrivial example being the form of degree 2:

*linear-quadratic family*.

As our main results reported below are asymptotic in nature and employ *N* as a big parameter, it is natural to ask how robust are our findings against relaxing the Gaussianity assumption. For the linear case Eq. (6) such generalization is possible, see e.g. [7], by considering rotationally-invariant ensemble of the associated matrices *A* and employing the large-*N* asymptotics of the so-called spherical integrals found in [24, 25]. Whether similar methods may help to settle the question for the linear-quadratic family by replacing the GOE matrices \(\mathcal{J}^{(k)}\) with their counterparts from more general class of rotationally-invariant random matrix ensembles remains an interesting topic for a future research.

### 1.2 Main Results

Our first main result is the following

### Proposition 1

*t*and

*Q*take values in intervals \([-R,R]\) and \([0,R^2]\), correspondingly, and \(w_s(u)\) is a non-decreasing function in \(u\in [R^2-Q,R^2]\). Then in the framework of the Parisi scheme of the Full Replica Symmetry Breaking (FRSB) the mean value of the parameter \(p_N\) characterising quality of the information recovery in the signal reconstruction scheme, Eqs. (2)–(5), with normally distributed noise \(\mathbf{b}\sim \mathcal{N}(\mathbf{0},\sigma ^2 \mathbf{1}_M)\) is given for \(N\rightarrow \infty \) by

*t*to be substituted to Eq. (15) should be found by simultaneously minimizing the functional \(\mathcal {E}[w_s(u);Q,v,t]\) over

*t*and maximizing it over all other parameters and the function \(w_s(u)\).

Our next result provides an explicit solution to this variational problem in a certain range of parameters.

### Proposition 2

*t*to the equation

*t*of Eq. (68) violates the inequality Eq. (69) the variational problem Eq. (14) can be solved by the FRSB Ansatz amounting to assuming the minimizer function \(w_s(u)\) to be

*continuous*and

*non-decreasing*. In this case the value

*p*of the quality parameter \(p_{\infty }\) definied in Eq. (15) is given by the solution of the following system of two equations in the variables \(p\in [0,1]\) and \(Q\in [0,R^2]\):

We finally note in passing that the FRSB scheme as we use it *automatically* includes the case of the so-called 1-Step Replica Symmetry Breaking (1-RSB) which corresponds to looking for a solution of the variational problem being a constant function \(w_s(u)=const, \, \forall u\in [R^2-Q,Q]\). Closer inspection, which we however omit, shows that such a solution respecting the required constraints on the parameters *v*, *t* and *Q* simply does not exist, and any nontrivial solution \(w_s(u)\) is continuous and increasing in the interval \([R^2-Q,Q]\). The question whether the variational problem may have a valid solution with higher discrete level of the replica-symmetry breaking (the so-called \(r-step\), which in our language will correspond to \(w_s(u)\) being nondecreasing in \([R^2-Q,Q]\), but with \(r-1\) discontinuities) has been however not investigated. Note, that in general in the spin-glass literature such a possibility is considered to be relatively exotic, though can not be excluded on general grounds, see [3].

#### 1.2.1 Results for the Linear-Quadratic Family of Encryptions

Both Propositions providing the solution of our reconstruction problem in full generality, for every specific choice of the covariance structure \(\Phi (u)\) the equations need to be further analyzed. In this work we performed a detailed analysis of the case of encryptions belonging to the linear-quadratic family Eq. (11) with the covariance structure of the form \(\Phi \left( u\right) =J_1^2\, u+\frac{1}{2}J_{2}^2u^2\). The most essential qualitative features of the analysis are summarized below.

*nonlinearity*in the encryption mapping. Our first result is that there exists a threshold value of this parameter, \(a=1\), such that for all encryptions in the family with \(a<1\) the variational problem is always solved with the Replica-Symmetric Ansatz Eq. (18). In contrast, for linear-quadratic encryptions with higher nonlinearity \(a>1\) there exists a threshold value of the ERP \(\mu =\frac{(a^{2/3}-a^{1/3}+1)^3}{a}:=\mu _{AT}(a)>1\) such that for any \(\mu \in (1,\mu _{AT}(a))\) the replica symmetric solution is broken in some interval of NSR and is replaced by one with FRSB. This implies that increasing redundancy for a fixed non-linearity one eventually always ends up in the replica-symmetric phase, see the phase diagram in Fig. 1.

In contrast, at a fixed nonlinearity \(a>1\) and not too big redundancy values \(\mu \in (1,\mu _{AT}(a))\) there exists generically an interval of scaled NSR’s \(\tilde{\gamma }^{(1)}_{AT}<\tilde{\gamma }<\tilde{\gamma }^{(2)}_{AT}\) such that the replica-symmetry is broken inside and preserved for \(\tilde{\gamma }\) outside that interval. The exact values \(\tilde{\gamma }_{AT}^{(1,2)}\) can be in general found only by numerically solving the 4th-order polynomial equation, see Eq. (57). At the same time, using that for large enough scaled NSR \(\tilde{\gamma }>\tilde{\gamma }^{(2)}_{AT}\) the replica symmetry is restored, one can employ the RS equation Eq. (57) to determine the behaviour of the quality parameter \(p_{\infty }(\tilde{\gamma })\) as \(\tilde{\gamma }\rightarrow \infty \). One finds that in all cases but one this quantity vanishes for asymptotically large values of NSR as \(p_{\infty }\sim \tilde{\gamma }^{-1/2}\), see Eq. (58), i.e. in qualitatively the same way as for purely linear system with \(a=0\).

The only exceptional case, showing qualitatively different behaviour to the above picture, is that of *purely quadratic* encryption with vanishing linear component,^{2} when \(J_1\rightarrow 0\) at a fixed value of \(J_2>0\). The appropriately rescaled NSR in this case is \(\hat{\gamma }=\frac{\gamma }{J_2^2R^2}\). In this limit the second threshold \(\tilde{\gamma }^{(2)}_{AT}\) escapes to infinity and the replica symmetry is broken for *all*\(\hat{\gamma }>\hat{\gamma }_{AT}=(\mu -1)^2/\mu \). Moreover, most importantly there exists a threshold NSR value \(\hat{\gamma }_c=\mu -\frac{1}{2}>\hat{\gamma }_{AT}\) such that \(p_{\infty }=0\) for \(\hat{\gamma }>\hat{\gamma }_c\) making the reconstruction impossible. The full curve \(p_{\infty }(\hat{\gamma })\) can be explicitly described in this case analytically. In particular, the behaviour close to the threshold NSR is given by \(p_{\infty }\sim (\hat{\gamma }_c-\hat{\gamma })^{3/4}\) and the non-trivial exponent 3 / 4 is completely controlled by the FRSB mechanism.

The existence of a sharp NSR threshold \(\hat{\gamma }_c\) in the pure quadratic encryption case may have useful consequences for security of transmitting the encrypted signal. Indeed, it is a quite common assumption that an eavesdropper may get access to the transmitted signal by a channel with inferior quality, characterized by higher level of noise. This may then result in impossibility for eavesdroppers to reconstruct the quadratically encoded signal even if the encoding algorithm is perfectly known to them.

#### 1.2.2 General Remarks on the Method

The task of optimizing various random ’cost functions’, not unlike \(\mathcal{H}_\mathbf{s}(\mathbf{x})\) in Eq. (7), is long known to be facilitated by a recourse to the methods of statistical mechanics, see e.g. [34] and [31] for early references and introduction to the method. In that framework one encounters the task of evaluating expected values over distributions of random variables coming through the cost function in both numerator and denominator of the equations describing the quantities of interest, see the right-hand side of Eq. (22) below. Performing such averaging is known to be one of the central technical problems in the theory of disordered systems. One of the most powerful, though non-rigorous, methods of dealing with this problem at the level of theoretical physics is the (in)famous replica trick, see [31] and references therein. A considerable progress achieved in the last decades in developing rigorous aspects of that theory [4, 33] makes this task, in principle, feasible for the cases when the random energy function \(\mathcal{H}_\mathbf{s}(\mathbf{x})\) is *Gaussian*-distributed. The model where configurations are restricted to the surface of a sphere are known in the spin-glass literature as ’spherical models’, but their successful treatment, originally nonrigorous [13, 14, 27] and in recent years rigorous[1, 5, 11, 12, 40, 41], seems again be restricted to the normally-distributed case. In the present [2] case however the cost function is *per se* not Gaussian, but represented as a sum of *squared* Gaussian-distributed terms. We are not aware of any systematic treatment of spherical spin glass models with such type of spin interaction. Some results obtained by extending replica trick treatment to this type of random functions were given by the present author in [18], but details were never published. To present the corresponding method on a meaningful example is one of the goals of the present paper. Indeed, we shall see that, with due modifications, the method is very efficient, and, when combined with the Parisi replica symmetry breaking Ansatz allows to get a reasonably detailed insight into the reconstruction problem. As squared gaussian-ditributed terms are common to many optimization problems based on the Least Square method, one may hope that the approach proposed in the present paper may prove to be of wider utility. In particular, an interesting direction of future research may be study of the minima, saddles and other structures of this type in the arising ’optimization landscape’ following an impressive recent progress in this direction for Gaussian spherical model, see [35] and references therein. This may help to devise better search algorithms for solutions of the optimization problems of this type.

Another technical aspect of our treatment which is worth mentioning is as follows. In problems of this sort replica treatment is much facilitated by noticing that after performing the disorder averaging the replicated partition function possesses a high degree of invariance: an arbitrary simultaneous *O*(*N*) rotation of all *n* replica vectors \(\mathbf{x}_a, \, a=1,\ldots ,n\) leaves the integrand invariant. To exploit such an invariance in the most efficient way one may use a method suggested in the framework of Random Matrix Theory in the works [17, 23]^{3} That method allowed one to convert the integrals over \(N-\) component vectors \(\mathbf{x}_{a},\, a=1,\ldots ,n\) to a single positive-definite \(n\times n\) matrix \(Q_{ab}\ge 0\). Such transformation than allows to represent the integrand in a form ideally suited for extracting the large-*N* asymptotic of the integral. In the context of spin glasses and related statistical mechanics systems this method was first successfully used in [22] and then [20, 21], and most recently in [29], and proved to be a very efficient framework for implementing the Parisi FRSB scheme. In the present problem however the integrand has lesser invariance due to presence of a fixed direction exemplified by the original message \(\mathbf{s}\). Namely, it is invariant only with respect to rotations forming a subgroup of *O*(*N*) consisting of all \(N\times N\) orthogonal transformations \(O_\mathbf{s}\) satisfying \(O_\mathbf{s}^{T}O_\mathbf{s}=\mathbf{1}_N\) and \(O_\mathbf{s}{} \mathbf{s}=\mathbf{s}\). In the Appendix C we prove a Theorem which is instrumental in adjusting our approach to the present case of a fixed direction. One may hope that this generalization may have other applications beyond the present problem.

## 2 Statistical Mechanics Approach to Reconstruction Problem

### 2.1 General Setting of the Problem

*N*spin variables \((x_1,\ldots , x_N)\), constrained to the sphere of radius \(|\mathbf{x}|=N\sqrt{R}\). This allows one to treat our minimization problem as a problem of Statistical Mechanics, by introducing the temperature parameter \(T>0\), and considering the Boltzmann-Gibbs weights \(\pi _{\beta }(\mathbf{x})=\mathcal{Z}_{\beta }^{-1} e^{-\beta \mathcal{H}_\mathbf{s}(\mathbf{x})}\) associated with any configuration \(\mathbf{x}\) on the sphere, with \(\mathcal{Z}_{\beta }\) being the partition function of the model for the inverse temperature \(\beta =T^{-1}\) :

### 2.2 Replica Trick

*n*before the limit as a positive integer. Employing Eq. (21) for each factor in \(\mathcal{Z}_{\beta }^{n-1}\) one than can represent Eq. (22) formally as

*O*(

*N*) consisting of all \(N\times N\) orthogonal transformations \(O_\mathbf{s}\) satisfying \(O_\mathbf{s}^{T}O_\mathbf{s}=\mathbf{1}_N\) and \(O_\mathbf{s}{} \mathbf{s}=\mathbf{s}\). Then the integrand in Eq. (28) remains invariant under a simultaneous change \(\mathbf{x}_a\rightarrow O_\mathbf{s} \mathbf{x}_a\) for all \(a=1,\ldots ,n\). In the Appendix C we prove a Theorem which is instrumental for implementing our previous approach to similar problems [22] to the present case of somewhat lesser invariance. Not surprisingly, in such a case the integration needs to go not only over \(n\times n\) matrix of scalar products \(Q_{ab}=(\mathbf{x}_a,\mathbf{x}_b)\ge 0\), but also over an

*n*-component vector \(\mathbf{t}=(t_1,\ldots ,t_n)\in \mathbb {R}^n\) of projections \(t_a=(\mathbf{x}_a,\mathbf{s})\). Applying the Theorem and rescaling for convenience the integration variables \(Q_{ab}\rightarrow NQ_{ab}\) and \(\mathbf{t}\rightarrow \sqrt{N} \mathbf{t}\) we bring Eq. (29) to the form

*N*and

*n*satisfying \(N>n+1\) and involved no approximations. Our goal is however to extract the leading behaviour of that object as \(N\gg 1\) and allowing formally

*n*to take non-integer values to be able to perform the replica limit \(n\rightarrow 0\).

### 2.3 Variational Problem in the Framework of the Parisi Ansatz

Clearly, the form of the integrand in Eq. (29) being proportional to the factor \(e^{-\frac{N}{2}F(Q,\mathbf{t})}\) is suggestive of using the Laplace (a.k.a saddle-point or steepest descent) method. In following this route we resort to a non-rigorous and heuristic, but computationally efficient scheme of Parisi replica symmetry breaking [31]. We implement this scheme in a particular variant most natural for models with rotational invariance, going back to Crisanti and Sommers paper [13], and somewhat better explained in the Appendix A of [22], and in even more detail in the Appendix C of [19]. We therefore won’t discuss the method itself in the present paper, only giving a brief account of necessary steps.

*Q*which for finite integer

*n*have a special hierarchically built structure characterized by the sequence of integers

*Q*matrix block-wise, and satisfying:

*n*diagonal entries \(Q_{aa}\) of the matrix

*Q*with one and the same value \(Q_{aa}=q_d:=q_{k+1}\ge q_k\). Note that in our particular case the diagonal entries \(q_d\) must in fact be chosen in the form \(q_d=R^2-t_a^2\), in order to respect the constraints impose by the integration domain Eq. (30). As to the vector \(\mathbf{t}\) of variables \(t_{a}\), we are making an additional assumption that with respect to those variables the integral is in fact dominated by equal values: \(t_a=t, \forall a=1,\ldots ,n\).

*Q*, with parameters \(m_l, \, l=0, \ldots , k+1\) shared by both matrices, but parameters \(q_l\) replaced by parameters \(g_l\) given by

*Q*those eigenvalues are listed, e.g., in the appendix C of [19], and for the matrix \(g(Q,\mathbf{t})\) the corresponding expressions can be obtained from those for

*Q*by replacing \(q_l\) by parameters \(g_l\) from Eq. (35). The subsequent treatment is much facilitated by introducing the following (generalized) function of the variable

*q*:

*x*(

*q*) is piecewise-constant non-increasing, and changes between \(x(0<q<q_0)=m_0\equiv n\) through \(x(q_{i-1}<q<q_i)=m_i\) for \(i=1, \ldots ,k\) to finally \(x(q_k<q<q_d)=m_{k+1}\equiv 1\). A clever observation by Crisanti and Sommers allows one to express eigenvalues of any function of the hierarchical matrix Q in terms of simple integrals involving

*x*(

*q*). In particular, for eigenvalues \(\lambda ^{(g)}_l\) of the matrix \(g(Q,\mathbf{t})\) we have:

*x*(

*q*) facilitates calculating quantities interesting to us in the replica limit as:

*x*(

*q*) is now transformed to a non-decreasing function of the variable

*q*in the interval \(q_0\le q \le q_k\), and satisfying outside that interval the following properties

*k*real parameters \(m_l\) described in Eq. (43) . Performing the corresponding limit and taking into account that in view of \(x(q<q_0))=0\) we have

*x*(

*q*) only its non-trivial part in the interval \(q_0,q_k\) we arrive to the following

### Proposition 2.1

*x*(

*q*) of the variable

*q*in the interval \(q_0\le q \le q_k\). Then in the framework of the replica trick the asymptotic mean value of the quality parameter \(\left\langle p^{(\beta )}_N\right\rangle \) as \(N\rightarrow \infty \) is given by

*t*is found by simultaneously minimizing the functional \(\phi [x(q);q_0,q_k,t]\) over

*t*and maximizing it over all other parameters and the function

*x*(

*q*).

*low temperature Ansatz*valid for \(T\rightarrow 0\)

*v*,

*Q*and \(w_s(u)\) tending to a well-defined finite limit as \(T\rightarrow 0\). Performing the corresponding limit in Eq. (47) and changing \(u\rightarrow u-t^2\) one arrives at the statement of the

**Proposition**1 in the Main Results section.

## 3 Analysis of the Variational Problem

*t*,

*Q*and

*v*. The conditions \(\frac{\partial {\mathcal {E}}}{\partial t}=0\) and \(\frac{\partial {\mathcal {E}}}{\partial Q}=0\) yield the two equations, the first being

### 3.1 Replica Symmetric Solution

*p*is nonvanishing for any value of the scaled NSR \(0\le \tilde{\gamma }<\infty \), and tends to zero as \(p\sim \sqrt{\mu /\tilde{\gamma }}\) for \(\tilde{\gamma }\gg 1\), in full agreement with the direct perturbation theory approach, see appendix A. For intermediate NSR values the solution can be easily plotted numerically, see Fig. 2.

*p*tends to zero as NSR \(\tilde{\gamma }\rightarrow \infty \) as in the purely linear case:

*p*for \(\gamma \rightarrow \infty \) is always given by Eq. (58), apart from the only limiting case of purely quadratic encryption, when \(a\rightarrow \infty \).

### 3.2 Solution with Fully Broken Replica Symmetry

*q*in that interval it yields the equation

*q*, and find that for any \(q\in [R^2-Q,R^2]\) holds:

We therefore conclude that the pair of equations Eqs. (62) and (67) is sufficient for finding the values of the parameters *t* and *Q*, and hence for determining the value of *p* giving the quality of the reconstruction procedure.

### 3.3 Analysis of Replica Symmetry Breaking for the Linear-Quadratic Family of Encryptions

In this section we use the following scaling variables naturally arising when performing the analysis of the general case of linear-quadratic family: the scaled NSR \(\tilde{\gamma }=\sigma ^2/J_1^2R^2\), the variables \(p=t/R\) and \(\tilde{Q}=Q/R^2\) and the non-linearity parameter \(a=(RJ_2/J_1)^2\).

#### 3.3.1 Position of the de-Almeida Thouless Boundary

*th*-order polynomial equation Eq. (71)

^{4}fully confirms the picture outlined above, giving the explicit criterion for existence of solutions in the \((\mu ,a)\) parameter plane, cf. Fig. 1:

- (i)
For a given \(\mu >1\) no solutions with \(p\in [0,1]\) are possible for \(a<1\), whereas for a fixed \(a>1\) no solutions exist for \(\mu >\mu _{AT}(a)\).

- (ii)
For \(\mu =\mu _{AT}(a)\) there exists a single solution: \(p=p_{AT}\).

- (iii)
For a given \(a>1\) and \(1<\mu <\mu _{AT}(a)\) there exist exactly two solutions \(0\le p_{2}< p_1 < 1\).

*p*given by solving the RS equation Eq. (57). In contrast, in the last case (iii) the two solutions give rise to two different AT thresholds in the scaled

*NRS*values: \(\tilde{\gamma }^{(2)}_{AT}>\tilde{\gamma }^{(1)}_{AT}\). In other words, for fixed values of parameters \(\mu \) and

*a*there is generically an interval of NSR’s \(\tilde{\gamma }^{(1)}_{AT}<\tilde{\gamma }<\tilde{\gamma }^{(2)}_{AT}\) such that the replica-symmetry is broken inside and preserved for \(\tilde{\gamma }\) outside that interval.

#### 3.3.2 Analysis of Solutions with the Broken Replica Symmetry for the Linear-Quadratic Family of Encryptions

After getting some understanding of the domain of parameters where replica symmetry is expected to be broken, let us analyse the pair of equations Eqs. (62) and (67), looking for a solution with \(0<Q\le 1\). As before introduce \(p=t/R\) and \(\tilde{Q}=Q/R^2\) as our main variables of interest.

We start with considering the two limiting cases in the family: that of purely linear scheme with \(\Phi (u)=J_1^2u\) and the opposite limiting case of purely quadratic encryption scheme with \(\Phi (u)=J_2^2\frac{u^2}{2}\). In the former case the previous analysis indicates that only RS solution must be possible. Indeed, we immediately notice that for purely linear scheme Eq. (67) takes the form \( p^2=\mu \) which can not have any solution as \(\mu > 1\) but \(p\in [0,1]\).^{5} We conclude that a solution with broken replica symmetry \(Q>0\) does not exist, so in this case the correct value of *p* is always given by solving the RS equation Eq. (55), as anticipated.

*p*given by Eq. (57) remains valid. For small \(0<\delta \ll 1\) one easily finds \(\tilde{Q}=\frac{\mu }{3(\mu -1})\delta +O(\delta ^2)\)

^{6}. On the other hand one can see that the solution \(\tilde{Q}(\delta )\rightarrow 1\) as \(\delta \rightarrow \delta _c=\frac{3}{2}-\frac{1}{\mu }\). so that a meaningful solution only exists in the interval \(\delta \in [0,\delta _c]\). Moreover, it is easy to show that for \(\delta \rightarrow \delta _c\) we have \(\tilde{Q}=1-\sqrt{\frac{2}{3}(\delta _c-\delta )}\). We see that the second of Eq. (78) then implies that when approaching the true threshold value \(\tilde{\gamma }^{(RSB)}_{c}=\mu -\frac{1}{2}\) dictated by broken replica symmetry the quality parameter vanishes as \(p\sim (\hat{\gamma }^{(RSB)}_{c}-\hat{\gamma })^{3/4}\) rather than as a square root, as in the replica-symmetric solution Eq. (56).

*p*from the second of Eq. (78), and combining it with RS expression Eq. (56) obtain the full corresponding curve for \(p(\hat{\gamma })\) for a given \(\mu \). In particular, for the above special value \(\mu =2\) the full curve can be described by an explicit expression:

*a*as long as \(\mu <\mu _{AT}(a)=\frac{(a^{2/3}-a^{1/3}+1)^3}{a}\) there exists two NSR thresholds \(\tilde{\gamma }_{AT}^{(1)}\) and \(\tilde{\gamma }_{AT}^{(2)}\) such that for \(\tilde{\gamma }\notin [\tilde{\gamma }_{AT}^{(1)}, \tilde{\gamma }_{AT}^{(2)}]\) the curve \(p(\tilde{\gamma })\) is given by RS solution Eq. (57), whereas for \(\tilde{\gamma }\in [\tilde{\gamma }_{AT}^{(1)}, \tilde{\gamma }_{AT}^{(2)}]\) the curve \(p(\tilde{\gamma })\) is given by the Full RSB solution from the system of two equations:

## Footnotes

- 1.
- 2.
It is worth noting that in the absence of linear component the encryption mapping \(V(\mathbf{s})\) in Eq. (11) becomes invariant with respect to the reflections \(\mathbf{s}\rightarrow -\mathbf{s}\). As a result, the least-square reconstruction may formally return solutions with negative values of the parameter \(p_N\) in Eq. (4). To avoid this we consider the pure quadratic case as the limit \(J_1\rightarrow 0\) taken

*after*\(N\rightarrow \infty \), which is enough to break the mentioned invariance. - 3.
- 4.
I am grateful to Dr. Mihail Poplavskyi for his help with the corresponding analysis.

- 5.
- 6.
The two other solutions of the cubic equation can be shown to be out of the interval (0, 1], see the explicit example below.

## Notes

### Acknowledgements

The author is grateful to Ali Bereyhi, Jean-Philippe Bouchaud, Ralf Mueller, Hermann Schulz-Baldes, Guilhem Semerjian, Nicolas Sourlas, Francesco Zamponi, and Lenka Zdeborova for enlightening discussions and encouraging interest in this work, to Christian Schmidt for careful reading of the manuscript and many advices, and to Dr. Mihail Poplavskyi for his help with analysis of Eq. (71) and preparing figures for this article. The financial support by EPSRC Grant EP/N009436/1 “The many faces of random characteristic polynomials” is acknowledged with thanks.

## References

- 1.Auffinger, A., Ben Arous, G.: Complexity of random smooth functions on the high-dimensional sphere. Ann. Prob.
**41**(6), 4214–4247 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 2.Auffinger, A.: Chen, Wei-Kuo: On the energy landscape of spherical spin glasses. Adv. Math.
**330**, 553–588 (2018)MathSciNetCrossRefzbMATHGoogle Scholar - 3.Auffinger, A., Zeng, Q.: Existence of two-step replica symmetry breaking for the spherical mixed p-spin glass at zero temperature. Commun. Math. Phys. (to appear)Google Scholar
- 4.Bovier, A.: Statistical Mechanics of Disordered systems: a Mathematical Perspective (Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge (2016)Google Scholar
- 5.Baik, J., Lee, J.O.: Fluctuations of the free energy of the spherical Sherrington-Kirkpatrick model. J. Stat. Phys.
**165**(2), 185–224 (2016)ADSMathSciNetCrossRefzbMATHGoogle Scholar - 6.Barbier, J., Krzakala, F., Macris, N., Miolane, L., Zdeborova, L.: Phase Transitions, Optimal Errors and Optimality of Message-Passing in Generalized Linear Models. arXiv:1708.03395
- 7.Bereyhi, A., Mueller, R., Schulz-Baldes, H.: Statistical Mechanics of MAP estimation: General replica Ansatz. arXiv:1612.01980
- 8.Bereyhi, A., Mueller, R., Schulz-Baldes, H.: Replica Symmetry Breaking in Compressive Sensing. IEEE Information Theory and Applications Workshop, (ITA) (2017), pp. 1–7 [e-preprint arXiv:1704.08013]
- 9.Bereyhi, A., Mueller, R.: Maximum-A-Posteriori signal recovery with prior information: applications to compressed sensing. arXiv:1802.05776
- 10.Blumensath, T.: Compressed sensing with nonlinear observations and related nonlinear optimization problems. IEEE Trans. Inf. Theory
**59**(6), 3466–3474 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 11.Chen, W.-K., Sen, A.: Parisi formula, disorder chaos and fluctuation for the ground state energy in the spherical mixed p-spin models. Commun. Math. Phys.
**350**, 129–173 (2017)ADSMathSciNetCrossRefzbMATHGoogle Scholar - 12.Chen, W.-K., Panchenko, D.: Temperature chaos in some spherical mixed p-spin models. J. Stat. Phys.
**166**(5), 1151–1162 (2017)ADSMathSciNetCrossRefzbMATHGoogle Scholar - 13.Crisanti, A., Sommers, H.-J.: The spherical p-spin interaction spin glass model: the statics. Zeitsch. f. Phys. B
**87**(3), 341–354 (1992)ADSGoogle Scholar - 14.Cugliandolo, L.F., Dean, D.S.: On the dynamics of a spherical spin-glass in a magnetic field. J. Phys. A
**28**, L453–459 (1995)ADSCrossRefzbMATHGoogle Scholar - 15.David, F., Duplantier, B., Guitter, E.: Renormalization theory for interacting crumpled manifolds. Nucl. Phys. B
**394**, 555–664 (1993)ADSMathSciNetCrossRefzbMATHGoogle Scholar - 16.de Almeida, J.R.L., Thouless, D.J.: Stability of the Sherrington-Kirkpatrick solution of a spin glass model. J. Phys. A
**11**(5), 983–990 (1978)ADSCrossRefGoogle Scholar - 17.Fyodorov, Y.V.: Negative moments of characteristic polynomials of random matrices: Ingham-Siegel integral as an alternative to Hubbard-Stratonovich transformation. Nucl. Phys. B
**621**, 643–674 (2002)ADSMathSciNetCrossRefzbMATHGoogle Scholar - 18.Fyodorov, Y.V.: On statistical mechanics of a single particle in high-dimensional random landscapes. Acta Phys. Pol. B
**38**(13), 4055–4066 (2007)ADSMathSciNetzbMATHGoogle Scholar - 19.Fyodorov, Y.V.: Multifractality and freezing phenomena in random energy landscapes: an introduction. Physica A
**389**, 4229–4254 (2010)ADSCrossRefGoogle Scholar - 20.Fyodorov, Y.V., Bouchaud, J.-P.: Statistical mechanics of a single particle in a multiscale random potential: Parisi landscapes in finite-dimensional Euclidean spaces. J. Phys. A
**41**, 324009 (2008)MathSciNetCrossRefzbMATHGoogle Scholar - 21.Fyodorov, Y.V., Le Doussal, P.: Topology trivialization and large deviations for the minimum in the simplest random optimization. J. Stat Phys.
**154**(1–2), 466–490 (2014)ADSMathSciNetCrossRefzbMATHGoogle Scholar - 22.Fyodorov, Y.V., Sommers, H.-J.: Classical particle in a box with random potential: exploiting rotational symmetry of replicated Hamiltonian. Nucl. Phys. B
**764**(3), 128–167 (2007)ADSMathSciNetCrossRefzbMATHGoogle Scholar - 23.Fyodorov, Y.V., Strahov, E.: Characteristic polynomials of random Hermitian matrices and Duistermaat-Heckman localisation on non-compact Kähler manifolds. Nucl. Phys. B
**630**, 453–491 (2002)ADSCrossRefzbMATHGoogle Scholar - 24.Guionnet, A., Zeitouni, O.: Large deviations asymptotics for spherical integrals. J. Funct. Anal.
**188**(2), 461–515 (2002)MathSciNetCrossRefzbMATHGoogle Scholar - 25.Guionnet, A., Maıda, M.: A Fourier view on the \(R\)-transform and related asymptotics of spherical integrals. J. Funct. Anal.
**222**(2), 435–490 (2005)MathSciNetCrossRefzbMATHGoogle Scholar - 26.Kogan, H., Marcus, M.B., Rosen, J.: Permanental processes. Commun. Stoch. Anal.
**5**(1), 6 (2011) arXiv:1008.3522 - 27.Kosterlitz, J.M., Thouless, D.J., Jones, R.C.: Spherical model of a spin glass. Phys. Rev. Lett.
**36**, 1217–1220 (1976)ADSCrossRefGoogle Scholar - 28.Krzakala, F., Mezard, M., Sausset, F., Sun, Y.F., Zdeborova, L.: Statistical-physics-based reconstruction in compressed sensing. Phys. Rev. X
**2**, 021005 (2012)Google Scholar - 29.Kurchan, J., Maimbourg, T., Zamponi, F.: Statics and dynamics of infinite-dimensional liquids and glasses: a parallel and compact derivation. J. Stat. Mech.
**2016**(3), 033210 (2016)MathSciNetCrossRefGoogle Scholar - 30.Marchenko, V.A., Pastur, L.A.: Distribution of eigenvalues for some sets of random matrices. Mat. Sb. (N.S.)
**72**(114), 507–536 (1967)MathSciNetGoogle Scholar - 31.Mezard, M., Parisi, G., Virasoro, M.: Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications. (World Scientific Lecture Notes In Physics). World Scientific Publishing Company, Singapore (1986)CrossRefGoogle Scholar
- 32.Montanari, A.: Statistical Estimation: from denoising to sparse regression and hidden cliques. In: Krzakala, F., et al. (eds.) Statistical Physics, Optimization, Inference, and Message-Passing Algorithms: Lecture Notes of the Les Houches School of Physics: Special Issue, October 2013. Oxford University Press, Oxford (2016)Google Scholar
- 33.Panchenko, D.: The Sherrington-Kirkpatrick Model. Springer Monographs in Mathematics. Springer, New York (2013)CrossRefzbMATHGoogle Scholar
- 34.Parisi, G.: Constraint optimization and statistical mechanics. In: book series: Proc. Int. Sch. Physics ENRICO FERMI
**155**, 205–228 (2004) [e-preprint arXiv:cs/0312011] - 35.Ros, V., Ben Arous, G., Biroli, G., Cammarota, C.: Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitions. arXiv:1804.02686
- 36.Percus, J.K.: Dimensional reduction of integrals of orthogonal invariants. Commun. Pure Appl. Math.
**40**(4), 449–453 (1957)MathSciNetCrossRefzbMATHGoogle Scholar - 37.Plan, Y., Vershynin, R.: The generalized lasso with nonlinear observations. IEEE Trans. Inf. Theory
**62**(3), 1528–1537 (2016)CrossRefzbMATHGoogle Scholar - 38.Schülke, C., Schniter, P., Zdeborova, L.: Phase diagram of matrix compressed sensing. Phys. Rev. E
**94**(6), 062136 (2016)ADSCrossRefGoogle Scholar - 39.Sourlas, N.: Spin-glass models as error-correcting codes. Nature
**339**, 693–695 (1989)ADSCrossRefGoogle Scholar - 40.Subag, E.: The complexity of spherical p-spin models—a second moment approach. Ann. Probab.
**45**(5), 3385–3450 (2017)MathSciNetCrossRefzbMATHGoogle Scholar - 41.Talagrand, M.: Free energy of the spherical mean-field model. Probab. Theory Relat. Fields
**134**, 339–382 (2006)MathSciNetCrossRefzbMATHGoogle Scholar - 42.Zdeborova, L., Krzakala, F.: Statistical physics of inference: Thresholds and algorithms. Adv. Phys.
**65**(5) 453–552 (2016) arXiv:1511.02476

## Copyright information

**OpenAccess**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.