Cryptanalysis of a public key cryptosystem based on Diophantine equations via weighted LLL reduction
Abstract
Researching postquantum cryptography is now an important task in cryptography. Although various candidates of postquantum cryptosystems (PQC) have been constructed, sizes of their public keys are large. Okumura constructed a candidate of PQC whose security is expected to be based on certain Diophantine equations (DEC). Okumura analysis suggests that DEC achieves the high security with small public key sizes. This paper proposes a polynomial timeattack on the oneway property of DEC. We reduce the security of DEC to finding special short lattice points of some lowrank lattices derived from public data. The usual LLL algorithm could not find the most important lattice point in our experiments because of certain properties of the lattice point. Our heuristic analysis leads us to using a variant of the LLL algorithm, called a weighted LLL algorithm by us. Our experiments suggest that DEC with 128 bit security becomes insecure by our attack.
Keywords
Weighted LLL reduction Publickey cryptosystem Postquantum cryptosystem Diophantine equationMathematics Subject Classification
94A60 11Y161 Introduction
Researching postquantum cryptography is now an important task in cryptography. In fact, National Institute of Standards and Technology published a draft of the report on postquantum cryptography NISTIR 8105 [23] (see also their announcement at PQCrypto 2016 [24]). Although various cryptosystems expected to be postquantum cryptosystems (PQC) have been already constructed, see [7, 11] for details, sizes of their public keys are large. Thus finding computationallyhard problems which allow us to construct PQC with public keys of small sizes is a very important task in cryptography.
A Diophantine problem is wellknown to be a computationallyhard problem in mathematics [12], and there are some cryptographic schemes based on the problem [6, 17, 20, 31], which are expected to have resistance to quantum algorithms. (Note that Diophantine problem here means a problem to find integral or rational zeros of a given multivariate polynomial with integer coefficients and high degree.) However, a polynomial timeattack on the oneway property of the cryptosystem [20] is proposed [10], and Proposition 2 in [17] suggests that the protocols [6, 17, 31] are impractical.
We can also consider the Diophantine problems over other rings. The Algebraic Surface Cryptosystem (ASC) [4] is based on the difficulty of the section finding problem, which can be viewed as the Diophantine problem over global function fields. Such the Diophantine problem is shown to be unsolvable in general [26, 29]. The security analysis suggests that ASC with public keys of sizes of about 500 bits achieves high security, see [4]. However, the ideal decomposition attack [15] breaks the oneway property of ASC.
Okumura [25] constructed a candidate of PQC of which the security is expected to be based on the difficulty of solving a special class of Diophantine equations, called Diophantine equations of degree increasing type, over \(\mathbb {Z}\) (we will recall a definition of a polynomial of degree increasing type in Sect. 3). We call this cryptosystem DEC for short. Okumura shows that the solvability of Diophantine equations of degree increasing type is undecidable in general, see Remark 3.2 of [25]. DEC is a number field analogue of ASC and use the twisted plaintext, obtained from a plaintext by using RSAlike modular arithmetic, and some random polynomials with large coefficients in the encryption process. These are the main ideas of DEC to resist the analogues of all attacks [15, 18, 28, 30] on ASC and cryptosystems [1, 2, 3], which are proposed previously as ASC. In Sect. 4 of [25], Okumura points out that the above ideas increase the number of possible parameters in DEC, and that breaking the oneway property of DEC will become infeasible. Okumura also points out that one can decode a plaintext correctly from the twisted plaintext by using polynomials of degree increasing type as public keys. We will review DEC and its recommended parameters briefly in Sect. 3.
Another important property of DEC in postquantum cryptography is that we may use public keys with small sizes, e.g., about 1, 200 bits with 128 bit security (see Remark 9). The size (1, 200 bits) is about 10 times smaller than sizes of public keys used in cryptosystems [21, 22, 27], which are wellknown to be efficient among the candidates of PQC, with 128 bit security. Thus we consider that the security analysis of DEC is an important task in cryptography.
1.1 Our contribution
In this paper, we propose a polynomial timeattack on DEC. We show a linearization technique to transform the oneway property of DEC to finding appropriate solutions of linear systems obtained from public data. The use of three polynomials as a ciphertext enables us to use the linearization technique which constructs linear systems. This is the first weakness of DEC. Our attack consists of three steps. In each step, we have a linear system and need to find its appropriate solution, i.e., we need to find an appropriate lattice point in the lattice which is the solution space of the linear system. We use a solution obtained in the first (resp. second) step to construct a linear system in the second (resp. third) step. After finding appropriate solutions of the linear systems in all the steps, it is possible to recover a plaintext with sufficiently high probability by applying the Babai nearest plane algorithm [5] and some modular arithmetic.
Our various experiments on our attack in Sect. 6 suggest that finding a correct solution results in breaking DEC with sufficiently high probability. More precisely, after we find a correct solution in the first step, we can solve the linear systems in the second and third steps (note that in the third step, we may use an incorrect solution obtained in the second step). Thus the success of the first step is most important for our attack.
The rank of the lattice occurring in the first step is low, e.g., 3rank in almost all cases, and a target lattice point in the first step is relatively short in the lattice. The quality of basis reduction algorithms such as the LLL algorithm [19] depends heavily on the rank of a lattice, and the LLL algorithm outputs a shortest lattice point in many cases for 3rank lattices, see [19]. Thus it seems that one can succeed in the first step by using the LLL algorithm (or other basis reduction algorithms). However, as we will see in Sect. 4.3, the usual LLL algorithm does not seem to work well for finding the target lattice point in the first step, where by the “usual LLL algorithm”, we mean the LLL algorithm in terms of pnorms (\(1 \le p \le \infty \)) \(\Vert {\varvec{ a}} \Vert _{p} := \left(  a_1 ^p + \ldots + a_n^p \right) ^{\frac{1}{p}}\). We heuristically analyse a reason why the usual LLL algorithm is not useful in our attack as follows: the target lattice point in the first step is not shortest, in terms of pnorms (\(1 \le p \le \infty )\), with high probability, but some of its entries are comparatively small. In other words, the target lattice point is a comparatively short (not neccessarily shortest) in terms of wellknown norms and has entries of unbalanced sizes.
1.2 Weighted LLL
We also note that using the weighted LLL algorithm can be also considered as using a rescaling of a lattice to find lattice points with entries of unbalanced sizes in an LLL reduced basis of the lattice. Such a method can be also found in Coppersmith’s method [9] (see also Chapter 19 of [16]) and in Faugére et al.’s method [14]. In our method, each entries of the weighted norm are 2power integers to use the knowledge of the bit length of entries of our target lattice point as in Faugére et al.’s method [14] (the possibility of knowing the bit length of entries of our target lattice point is the second weakness of DEC).
1.3 Experimental verification of our attack
Our many experiments in Sect. 6 suggest that the weighted LLL algorithm can find target lattice points in the first step of our attack with high probability (the probability being about from 70 to 90%) for the recommended parameters in Sect. 3. These results suggest that the weighted LLL algorithm is effective in cryptanalysis of cryptosystems whose security are reduced to finding lattice points with special properties: they are not shortest, but the bit length of their entries are almost known and comparatively small among entries of lattice points in certain lattices. In addition, our experiments also suggest that our attack breaks the oneway property of DEC with probability being about from 20 to 40% (this probability is sufficient in practical cryptanalysis). Our detailed complexity analysis on our attack and our experiments show that our attack is performed in polynomial time, and thus we conclude that our attack via the weighted LLL algorithm is practical and makes DEC insecure.
This paper is organized as follows: In Sect. 2, we give a definition of a weighted norm and describe the weighted LLL algorithm. In Sect. 3, we give a brief review of DEC. In Sect. 4, we describe the outline and some assumptions of our attack, and we also give an algorithm of our attack and a toy example to illustrate our attack. In Sect. 5, we analyse the complexity on our attack. In Sect. 6, we give some experimental results on our attack.
1.4 Notation
An mdimensional lattice is defined as a discrete additive subgroup of an mdimensional vector space over \(\mathbb {R}\). It is wellknown that for any lattice \(\mathcal{L}\), there exist \(\mathbb {R}\)linearly independent vectors generating \(\mathcal{L}\) as a \(\mathbb {Z}\)module. The rank of \(\mathcal{L}\) is its rank as a \(\mathbb {Z}\)module. For any lattice in \(\mathbb {R}^m\) and its basis \(\{ {\varvec{b}}_1, \ldots , {\varvec{b}}_r \}\), let \({\varvec{U}}\) be an \(r \times m\) matrix whose ith row vector coincides with \({\varvec{b}}_i\) for each i. Then we call \({\varvec{U}}\) the basis matrix of the lattice. Let \(\langle \cdot , \cdot \rangle : \mathbb {R}^n \times \mathbb {R}^n \rightarrow \mathbb {R}\) be the natural inner product for some \(n \in \mathbb {Z}_{>0}\). For a vector \({\varvec{v}} \in \mathbb {R}^n\), we denote the Euclidean norm of \({\varvec{v}}\) by \(\Vert {\varvec{v}} \Vert \). We define the rounding function \(\lfloor \cdot \rceil : \mathbb {R} \rightarrow \mathbb {Z}\) as \(\lfloor c \rceil := \lfloor c + \frac{1}{2} \rfloor \) for any \(c \in \mathbb {R}\). Let \({\varvec{M}}\) be an \(m \times n\) matrix over \(\mathbb {Z}\) and \(\varphi _{{\varvec{M}}}\) the homomorphism as additive groups between \(\mathbb {Z}^{m} \rightarrow \mathbb {Z}^{n}\) defined by \({\varvec{v}} \mapsto {\varvec{v}} {\varvec{M}}\). Then the kernel of \(\varphi _{{\varvec{M}}}\) is a lattice in \(\mathbb {R}^m\), and we call it the kernel lattice of \({\varvec{M}}\).
2 The weighted LLL algorithm
In this section, we explain the weighted LLL algorithm, which is a key of our attack in Sect. 4, briefly. First, we define a weighted norm and a weighted lattice. They are useful for describing the weighted LLL algorithm.
Definition 1
For a lattice \(\mathcal{L} \subset \mathbb {R}^m\) and a vector \({\varvec{w}}= \left( w_1, \ldots , w_m \right) \in {\left( \mathbb {R}_{>0} \right) }^m\), set a diagonal matrix \({\varvec{W}}\) whose (i, i)entry is \(w_i\) for \(1 \le i \le m\). We consider the isomorphism \(f_{{\varvec{W}}} : \mathbb {R}^m \longrightarrow \mathbb {R}^m\) by \({\varvec{x}} \mapsto {\varvec{x}} {\varvec{W}}\). Then, it is easy to show the equivalence of finding shortest lattice points, related with each other, in two lattices \(\mathcal{L}^{\varvec{w}}\) and \(f_{{\varvec{W}}} \left( \mathcal{L} \right) \).
The weighted LLL algorithm for \({\varvec{w}}\) is an algorithm to compute an LLL reduced basis (with respect to \(\Vert \cdot \Vert _{\varvec{w}}\)) of \(\mathcal{L}^{\varvec{w}}\) (we call such a basis a weighted LLL reduced basis for \({\varvec{w}}\) in this paper).
The most important lattice point in our attack is not necessarily shortest in a lowrank lattice, but only some of its entries are comparatively small. This property leads us to applying the weighted LLL algorithm to find such a lattice point by carefully controlling the entries of a weighted LLL reduced basis, see Sect. 4.3.
Remark 2
Controlling the entries of a basis output by the LLL algorithm is used in Coppersmith’s method [9] and Faugére et al.’s method [14], see also Chapter 19 of [16]. In their method, the scale of a lattice (or equivalently an inner product used in the LLL algorithm) is changed by heuristic ways. One can conduct such changes by changing a norm from the Euclidean norm to a weighted norm for some weight. In particular, our method for choosing a weighted norm is the same as the method in [14], see Step 12 of our algorithm in Sect. 4.2.
3 Brief review of DEC
In this section, we review DEC briefly, see Sect. 3 in [25] for details. As we mentioned in Sect. 1, DEC is constructed as a candidate of PQC and has the property, which is strongly desired in postquantum cryptography, that sizes of public keys in DEC is small, e.g., about 1, 200 bits with 128 bit security, see Remark 9. Note that sizes of public keys in cryptosystems [21, 22, 27], which are wellknown to be efficient among the candidates of PQC, are about 10 times larger than 1, 200 bits.
3.1 Definiton of polynomials of degree increasing type
Definition 3
Remark 4
 (1)
From Definition 3, it is easy to see that \(X \left( \underline{x} \right) \) is of degree increasing type if and only if the total degrees of the monomials of \(X \left( \underline{x} \right) \) are different each other.
 (2)
Let X be a polynomial of degree increasing type. By the following order \(\succ \), the support \(\varLambda _X\) becomes a totally ordered set: for two elements \(\left( i_1, \ldots , i_n \right) \) and \(\left( j_1, \ldots , j_n \right) \) in \(\varLambda _X\), we have \(\left( i_1, \ldots , i_n \right) \succ \left( j_1, \ldots , j_n \right) \) if and only if \(i_1 + \cdots + i_n > j_1 + \cdots + j_n\).
Throughout this paper, whenever a polynomial X is of degree increasing type, we endow \(\varLambda _X\) with the total order given in Remark 4 (2).
Example 5
The polynomial \(X \left( x, y, z \right) := 3 x^3 y^2 z  4 x^2 y^2  x y z +5 y z + y + 11 \in \mathbb {Z}[x, y, z]\) is of degree increasing type.
Now, we describe DEC according to [25]. Note that Okumura did not suggest the security parameter because his purpose was to design the encryption scheme with 128 bit security. However, we here set the security parameter \(\lambda \) to analyse the complexity of our attack for each security level.
In accordance with [25], we regard the total degree of a public key polynomial as a parameter, which we denote by \(w_X\). Note that the parameter \(w_X\) is taken to be an integer independent of the security parameter \(\lambda \). In Remark 7 below, we will describe the reason why DEC has the two independent parameters \(\lambda \) and \(w_X\).
3.2 Key generation process

Secret Key: A vector \(\underline{{\varvec{a}}} := \left( a_1, \ldots , a_n \right) \in \mathbb {Z}^n\).
 Public Key:
 (1)
A positive integer d with \(\mathrm{gcd} \left( a_i, d \right) = 1\) for all \(1 \le i \le n\).
 (2)
A positive integer e with \(\mathrm{gcd} \left( e, \varphi \left( d \right) \right) = 1\), where \(\varphi \) is the Euler function.
 (3)
A polynomial \(X \left( \underline{x} \right) \in \mathbb {Z}[ \underline{x} ]\) of degree increasing type such that X is irreducible, \(X \left( \underline{{\varvec{a}}}/d \right) = 0\) and \(\# \varLambda _X \le w_X\), where \(\varLambda _X\) and \(w_X\) denote the support and the total degree of X, respectively.
 (1)
 Construction of \(X \left( \underline{x} \right) \):
 (1)
Choose \(\varLambda \subset {\left( \mathbb {Z}_{\ge 0} \right) }^n\) such that \(3 \le \# \{ \sum \underline{{\varvec{i}}} \ ; \ \underline{{\varvec{i}}} \in \varLambda \} = \# \varLambda < \infty \) and \(\underline{{\varvec{0}}} \in \varLambda \), where \(\underline{{\varvec{0}}}:= \left( 0, \ldots , 0 \right) \in {\left( \mathbb {Z}_{\ge 0} \right) }^n\).
 (2)
Let \(\underline{{\varvec{k}}}\) denote the maximal element of \(\varLambda \) (note that \(\varLambda \) is a totally ordered set in terms of the order given in Remark 4 (2)). Choose a random nonzero integer \(c_{\underline{{\varvec{i}}}}\) for each \(\underline{{\varvec{i}}} \in \varLambda \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}\). For a choice of \(c_{\underline{{\varvec{i}}}}\), see Remark 9 (2).
 (3)Choose random integers \(c_{\underline{{\varvec{k}}}}\) and \(c_{\underline{{\varvec{0}}}}\) such thatwhere \(w:= \max \{ \sum \underline{{\varvec{i}}} \ ; \ \underline{{\varvec{i}}} \in \varLambda \}\).$$\begin{aligned} c_{\underline{{\varvec{k}}}} {\underline{{\varvec{a}}}}^{\underline{{\varvec{k}}}} + c_{\underline{{\varvec{0}}}} d^{w}= & {}  \sum _{{\varvec{k}} \in \varLambda \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}} c_{\underline{{\varvec{i}}}} {\underline{{\varvec{a}}}}^{\underline{{\varvec{i}}}} d^{w  \sum \underline{{\varvec{i}}}}, \end{aligned}$$(1)
 (4)
Set \(\varLambda _X := \varLambda \) and \(X \left( \underline{x} \right) := \sum _{\underline{{\varvec{i}}} \in \varLambda _X} c_{\underline{{\varvec{i}}}} \underline{x}^{\underline{\varvec{i}}}\).
 (1)
Remark 6
There exist integers \(c_{\underline{{\varvec{k}}}}\) and \(c_{\underline{{\varvec{0}}}}\) such that the equality (1) is satisfied because \(a_i\) and d are mutually prime for each \(i \in \{ 1, \ldots , n \}\) from the assumption.
Remark 7
DEC has two parameters \(\lambda \) and \(w_X\) for the following reason: The public key of DEC is a Diophantine equation X of degree increasing type, and the secret key is its solution. Since there is no algorithm for solving Diophantine equations of degree increasing type, we set the security parameter, denoted by \(\lambda \), which determines the security level against the key recovery attack by the brute force search (note that \(\lambda \) also determines the security level against some attacks on the oneway property of DEC, see [25]). On the other hand, \(w_x\) is an important parameter which complicates public diophantine equations and makes solving them difficult (by any method other than the brute force search), see also Remark 9.
3.3 Encryption process
 Plaintext: A polynomial \(m \in \mathbb {Z}[ x_1, \ldots , x_n ]\) such that
 (a)
\(\varLambda _m = \varLambda _X\),
 (b)
\(1< c_{i_1, \ldots , i_n} \left( m \right) < d\) for all \(\left( i_1, \ldots , i_n \right) \in \varLambda _m\),
 (c)
\(\gcd \left( c_{i_1, \ldots , i_n} \left( m \right) , d \right) = 1\) for all \(\left( i_1, \ldots , i_n \right) \in \varLambda _m\).
 (a)
 Encryption Process:
 (1)
Choose a positive integer \(N \in \mathbb {Z}_{>0}\) uniformly so that we have \(N d > 2^{\lambda } H \left( X \right) \). For a size of N, see Section 3.5 below.
 (2)
Construct \(\widetilde{m} \left( \underline{x} \right) \in \mathbb {Z}[\underline{x}]\), called the twisted plaintext, by setting \(\varLambda _{\widetilde{m}} := \varLambda _{m}\) and \(c_{\underline{{\varvec{i}}}} \left( \widetilde{m} \right) := {c_{\underline{{\varvec{i}}}} \left( m \right) }^e \left( \mathrm{mod} \ N d \right) \), where \(0< c_{\underline{{\varvec{i}}}} \left( \widetilde{m} \right) < N d\) for \(\underline{{\varvec{i}}} \in \varLambda _{\widetilde{m}}\).
 (3)Choose \(f \left( \underline{x} \right) \in \mathbb {Z}[ \underline{x} ]\) uniformly at random such that
 (a)
\(\varLambda _f = \varLambda _X\),
 (b)
\(H \left( \widetilde{m} \right)< c_{\underline{{\varvec{k}}}} \left( f \right) < N d\) and \(\mathrm{gcd} \left( c_{\underline{{\varvec{k}}}} \left( f \right) , d \right) = 1\), where \(\underline{{\varvec{k}}}\) denotes the maximal element of \(\varLambda _f\).
 (a)
 (4)
Choose \(s_{j} \left( \underline{x} \right) , r_{j} \left( \underline{x} \right) \in \mathbb {Z}[\underline{x}]\) uniformly at random so that we have \(\varGamma _{s_j} = \varGamma _{X}\) and \(\varGamma _{r_j} = \varGamma _f\) for \(1 \le j \le 3\).
 (5)
Put \(F_j \left( \underline{x} \right) := \widetilde{m} \left( \underline{x} \right) + s_j \left( \underline{x} \right) f \left( \underline{x} \right) + r_j \left( \underline{x} \right) X \left( \underline{x} \right) \) for \(1 \le j \le 3\). Send \(\left( F_1, F_2, F_3, N \right) \) as a ciphertext.
 (1)
3.4 Decryption process
 Decryption Process:
 (1)By substituting \(\underline{{\varvec{a}}} / d\), a zero of \(X \left( \underline{x} \right) \), into \(F_j \left( \underline{x} \right) \), we obtainCompute$$\begin{aligned} h_j := F_j \left( \underline{{\varvec{a}}}/d \right) = \widetilde{m} \left( \underline{{\varvec{a}}}/d \right) + s_j \left( \underline{{\varvec{a}}}/d \right) f \left( \underline{{\varvec{a}}}/d \right) \text{ for } 1 \le j \le 3. \end{aligned}$$$$\begin{aligned} H_1 := \left( h_1  h_2 \right) d^{2 w_X}= & {} \left( s_1 \left( \underline{{\varvec{a}}}/d \right)  s_2 \left( \underline{{\varvec{a}}}/d \right) \right) f \left( \underline{{\varvec{a}}}/d \right) d^{2 w_X}, \\ H_2 := \left( h_1  h_3 \right) d^{2 w_X}= & {} \left( s_1 \left( \underline{{\varvec{a}}}/d \right)  s_3 \left( \underline{{\varvec{a}}}/d \right) \right) f \left( \underline{{\varvec{a}}}/d \right) d^{2 w_X}. \end{aligned}$$
 (2)
Compute \(g := \mathrm{gcd} \left( H_1, H_2 \right) \). If \(\mathrm{gcd} \left( g, d \right) > 1\), then let \(d'\) be the smallest factor of g satisfying \(\mathrm{gcd} \left( d, g/{d'} \right) = 1\) and replace g by \(g/{d'}\).
 (3)
Compute \(H := h_1 d^{2 w_X} \left( \mathrm{mod} \ g \right) \) and \(\mu := H d^{ w_X} \left( \mathrm{mod} \ g \right) \).
 (4)
Obtain the plaintext polynomial \(m \left( \underline{x} \right) \) from \(\mu \) or \(\mu  g\) by using an algorithm described in Sects. 3.4 and 3.5 of [25].
 (1)
3.5 Parameter size
 (1)The sizes of \(\underline{{\varvec{a}}}\), d, e and N:$$\begin{aligned}&2^{\frac{\lambda }{2}} \le d< 2^{\frac{\lambda }{2}+1}, \ (\lambda + 1) + \left( \frac{\lambda }{2} + 1\right) w_X \le e< 2 \left( (\lambda + 1) + \left( \frac{\lambda }{2} + 1\right) w_X \right) , \\&\frac{2^{\left\lceil \frac{\lambda }{n1} \right\rceil }}{\varphi \left( d \right) } d \le  a_i < \frac{2^{\left\lceil \frac{\lambda }{n1} \right\rceil +1}}{\varphi \left( d \right) } d \left( 1 \le i \le n \right) , \\&2^{\lambda + \left( \frac{\lambda }{2} + 1\right) \left( w_X  1 \right) } \le N < 2^{\lambda + 1 + \left( \frac{\lambda }{2} + 1\right) \left( w_X  1 \right) }. \end{aligned}$$
 (2)The size of a secret key is at mostbits.$$\begin{aligned} \left( \left\lceil \frac{\lambda }{n1} \right\rceil + 1 \right) n + \lceil \mathrm{log}_2 \ d  \mathrm{log}_2 \ \varphi \left( d \right) \rceil \end{aligned}$$
 (3)The size of a public key is at mostbits.$$\begin{aligned} \left( \left\lceil \frac{\lambda }{n1} \right\rceil + \left( \frac{\lambda }{2} + 2 + b\right) + \lceil \mathrm{log}_2 \ d  \mathrm{log}_2 \ \varphi \left( d \right) \rceil \right) w_X + (\lambda + 1) + \lceil \mathrm{log}_2 \ e \rceil \end{aligned}$$
 (4)The size of a ciphertext is at mostbits. Note that the size of each coefficient of \(F_i\) is at most$$\begin{aligned} \frac{3}{2} \left( w_X^2 + w_X \right) \left( \lambda + 1 + (\lambda + 2) w_X + \lceil \mathrm{log}_2 \ w_X \rceil \right) + \lambda + 1 + \left( \frac{\lambda }{2} + 1\right) \left( w_X  1 \right) \end{aligned}$$bits for \(i = 1\), 2, and 3.$$\begin{aligned} \lambda + 1 + (\lambda + 2) w_X + \lceil \mathrm{log}_2 \ w_X \rceil \end{aligned}$$
Remark 9
 (1)
In Sect. 4.5 of [25], it is pointed out that we should use a polynomial X satisfying \(w_X \ge 5\), \(n \ge 3\) and some conditions as a public key in order to avoid finding rational solutions to \(X = 0\). However, polynomials of degree increasing type are in a special class of polynomials, and finding rational zeros of such polynomials may be easier than finding those of general polynomials. Moreover, although finding rational zeros of polynomials of higher degree seems to be difficult in general, we should consider sizes of public keys and ciphertexts. Thus we recommend to use X of degree 10 as a public key.
 (2)
In Sect. 5 of [25], it is pointed out that for a public key X and \(\underline{{\varvec{i}}} \in \varLambda _X \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}\), we may choose \(c_{\underline{{\varvec{i}}}}(X) \le 2^{10}\), where \(\underline{{\varvec{k}}}\) denotes the maximal element of \(\varLambda _X\). However, since solving Diophantine equations of degree increasing type may be easier than solving more general Diophantine equations as we mentioned above, we should also consider using larger \(c_{\underline{{\varvec{i}}}}(X)\) for \(\underline{{\varvec{i}}} \in \varLambda _X \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}\) to deal with a wide class of polynomials of degree increasing type. In our experiments of Sect. 6, we choose \(c_{\underline{{\varvec{i}}}}(X)\) so that the sizes of \(\left c_{\underline{{\varvec{i}}}}(X) \right \) are b bits for \(b = 10, 50\) and 100.
 (3)
When \(\lambda = 128\), \(w_X = \# \varLambda _X\) and \(b = 10\), we generated 100 public keys X randomly and measured their sizes. As a result, their average size is about 1, 200 bits. This size (1, 200 bits) is about 10 times smaller than sizes of public keys in cryptosystems [21, 22, 27], which are wellknown to be efficient among the candidates of PQC.
3.6 Toy example of DEC

Secret Key: \(\underline{{\varvec{a}}} = \left( a, b \right) = \left( 47, 49 \right) \in \mathbb {Z}^2\).

Public Key: \((d, e, X) = (5, 17, 125 x^3 + 675 y  110438)\).
(\(\varLambda _X = \{ \left( 3, 0 \right) , \left( 0,1 \right) , \left( 0, 0 \right) \}\), \(\underline{{\varvec{k}}} = \left( 3, 0 \right) \), \(H \left( X \right) =110438\).)

Plaintext: \(m \left( \underline{x} \right) = m \left( x, y \right) = 3 x^3 + 3 y + 2\).
 Objects for Encryption:
 (1)
\(N = 353408\) (\(N d =1767040\)).
 (2)
\(\widetilde{m} \left( \underline{x} \right) = \widetilde{m} \left( x, y \right) = 146243 x^3 + 146243 y + 131072\) (\(H \left( \widetilde{m} \right) = 146243\)).
 (3)
\(f \left( \underline{x} \right) = f \left( x, y \right) = 949843 x^3 + 1324952 y + 1109775\).
(\(c_{\underline{{\varvec{k}}}} \left( f \right) = 949843\), \(H \left( \widetilde{m} \right) = 146243< c_{\underline{{\varvec{k}}}} \left( f \right) = 949843 < 1767040 = N d\).)
 (4)\(s_j\) and \(r_j\):$$\begin{aligned} s_1= & {} 115 x^3 + 924 y + 126337, \ \ s_2 = 82 x^3 + 962 y + 89939, \\ s_3= & {} 67 x^3 + 977 y + 121816, \ \ r_1 = 691019 x^3 + 1363650 y + 1329029, \\ r_2= & {} 852655 x^3 + 1584164 y + 2007688, \\ r_3= & {} 940020 x^3 + 2016302 y + 1144882. \end{aligned}$$
 (5)Cipher Polynomials: \(F_j := \widetilde{m} + s_j f + r_j X\).$$\begin{aligned} F_1= & {} 195609320 x^6 + 1666918487 x^3 y + 43979457762 x^3 + 2144719398 y^2 \\&+ 18714355042 y  6569529455, \\ F_2= & {} 184469001 x^6 + 1795957655 x^3 y  8395474520 x^3 + 2343914524 y^2 \\& 53364106711 y  121912862547, \\ F_3= & {} 181141981 x^6 + 1903319645 x^3 y + 12109757546 x^3 + 2655481954 y^2 \\& 59418815676 y + 8750004156. \end{aligned}$$
 (1)
4 Weighted LLLbased polynomial timeattack for DEC
We give in this section our attack algorithm against DEC, based on the weighted LLL. We use the following notation described in Notation of Sect. 1: for a polynomial \(h = \sum _{\underline{{\varvec{i}}} \in \varLambda _h}c_{\underline{{\varvec{i}}}}(h)x^{\underline{{\varvec{i}}}} \in \mathbb {Z}[ \underline{x} ]\), let \({\varvec{h}} := \left( c_{\underline{{\varvec{i}}}_1}(h), \ldots , c_{\underline{{\varvec{i}}}_{\# \varLambda _h}}(h) \right) \), where \(\varLambda _h = \{ \underline{{\varvec{i}}}_1,\ldots , \underline{{\varvec{i}}}_{\# \varLambda _h} \}\) is the support of h. Note that \(\varLambda _h\) is an ordered set (see Notation in Sect. 1).
Let \((d, e, X(\underline{x}) ) \in \mathbb {Z}^2 \times ( \mathbb {Z}[ \underline{x} ] )\) and \(( F_1(\underline{x}), F_2(\underline{x}), F_3(\underline{x}), N ) \in ( \mathbb {Z}[ \underline{x} ] )^3 \times \mathbb {Z}\) be a public key and a ciphertext, as described in Sects. 3.2 and 3.3. Let \(m(\underline{x}) \in \mathbb {Z}[ \underline{x} ]\) be a plaintext. Each cipher polynomial is of the form \(F_j (\underline{x}) = \widetilde{m}(\underline{x}) + s_j(\underline{x}) f(\underline{x}) + r_j(\underline{x}) X(\underline{x})\) for the twisted plaintext \(\widetilde{m}(\underline{x})\) and some random polynomials \(f (\underline{x})\), \(s_j (\underline{x})\) and \(r_j (\underline{x})\). For the choice of \(f (\underline{x})\), \(s_j (\underline{x})\) and \(r_j (\underline{x})\), see Sect. 3.3 for details. We write \(\varLambda _X = \{ \underline{{\varvec{i}}}_1, \ldots , \underline{{\varvec{i}}}_q \}\) with \(\underline{{\varvec{i}}}_1 \succ \cdots \succ \underline{{\varvec{i}}}_q\), where the total order \(\succ \) on \(\varLambda _X\) is given in Remark 4 (2). Recall that the supports of m, \(\widetilde{m}\), \(s_j\), \(r_j\) and f (\(1 \le j \le 3\)) are the same as \(\varLambda _X\), which allows attackers to suppose \(\varLambda _{F_1} = \varLambda _{F_2} = \varLambda _{F_3}\). Let \(\underline{{\varvec{k}}}\) denote the maximal element of \(\varLambda _X\). To simplify the notation, put \(q := \# \varLambda _X\) throughout this section. For recovering m, it suffices to get the correct \(\widetilde{m}\).
4.1 Idea of our attack
4.1.1 First step: determination of \(s^{\prime }_j\) for \(j = 1\) and 2
Here, we describe how to determine \({\varvec{s}}^{\prime }_j\) for \(j = 1\) and 2. (As we mentioned in Sect. 1, the vectors \({\varvec{s}}^{\prime }_j\) (\(j = 1\) and 2) are the most important target vectors). In the equality (4), we regard the coefficients of \(s^{\prime }_j \left( \underline{x} \right) \) and \(g \left( \underline{x} \right) \) as indeterminates. We then obtain the linear system \({\varvec{u}} {\varvec{A}} = {\varvec{0}}\), where \({\varvec{A}}\) is the \(\left( \left( 2 q + \# \varLambda _{X^2} \right) \times \# \varLambda _{X^3} \right) \) coefficient matrix of the linear system. We denote by \(\mathcal{L}^{\prime }_1\) the kernel lattice of \({\varvec{A}}\), where the kernel lattice of \({\varvec{A}}\) is defined as the nullspace of \({\varvec{A}}\), see Notation in Sect. 1. Let \(\mathcal{L}_1\) be the lattice spanned by the vectors consisting of the 1(2q)th entries of the elements in \(\mathcal{L}^{\prime }_1\). Experimentally, the rank of \(\mathcal{L}_1\) is equal to 3 in many cases, see Remark 27 in Sect. 6. Thus, we assume the following condition:
Assumption 10
The rank of \(\mathcal{L}_1\) is equal to 3.
Assumption 11
The \(\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right) \) is a shortest vector in \(\mathcal{L}_1^{\varvec{w}}\).
Let \(f_{{\varvec{W}}}\) be the isomorphism described in Sect. 2 from \(\mathbb {R}^{2q}\) to \(\mathbb {R}^{2q}\) as \(\mathbb {R}\)vector spaces. From Assumption 10, the rank of \(f_{{\varvec{W}}}(\mathcal{L}_1)\) is equal to 3. This means that we can expect the weighted LLL reduction for the weight \({\varvec{w}}\) to output a shortest vector in \(\mathcal{L}_1^{\varvec{w}}\) with high probability. Thus it is expected to find the correct \(\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right) \) via the weighted LLL reduction for the weight \({\varvec{w}}\), see Sect. 2 and Assumption 11.
Remark 12
As we will see in Sect. 4.3, one may fail in determining \(( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 )\) even if one adopts the LLL reduction in terms of the pnorm (\(1 \le p \le \infty \)) as a lattice reduction for \(\mathcal{L}_1\). Thus, the above assumptions and applying the weighted LLL reduction to \(\mathcal{L}_1\) are crucial for our attack.
4.1.2 Second step: obtaining a candidate of f
Here, we describe how to determine a candidate of f. We substitute \(s^{\prime }_1 \left( \underline{x} \right) \) and \(s^{\prime }_2 \left( \underline{x} \right) \) obtained in Step 1 into (2) and (3). In a similar way to Step 1, by regarding the coefficients of \(f \left( \underline{x} \right) \) and \(r^{\prime }_j \left( \underline{x} \right) \) for \(j=1\) and 2 as indeterminates, we have the linear system. We then fix \(f^{\prime } \left( \underline{x} \right) \) such that (2) and (3) hold and that \(f^{\prime } \left( \underline{x} \right) \) is close to the correct \(f \left( \underline{x} \right) \), i.e., the absolute values of all coefficients of the polynomial \(f^{\prime } \left( \underline{x} \right)  f \left( \underline{x} \right) \) are small. Note that \(f^{\prime } \left( \underline{x} \right) \) does not necessarily coincide with the correct \(f \left( \underline{x} \right) \) to recover \(\widetilde{m}\) (cf. Remark 18 and Steps 33 and 34 in Sect. 4.2).
Remark 13
4.1.3 Third step: recovery of \(\widetilde{m}\)
Assumption 14
The rank of \(\mathcal{L}_3\) is equal to 3.
Let \({\varvec{w}}_0\) be one solution to \({\varvec{w}} {\varvec{C}} = {\varvec{c}}\) and \(\{ {\varvec{w}}_1, {\varvec{w}}_2, {\varvec{w}}_3 \}\) a basis of \(\mathcal{L}_3\). Note that every integral solution to the system is represented as \({\varvec{w}}_0 + a_1 {\varvec{w}}_1 + a_2 {\varvec{w}}_2 + a_3 {\varvec{w}}_3\) (\(a_i \in \mathbb {Z}\), \(i=1,2\) and 3). The 1\(\# \varLambda _X\)th entries of \({\varvec{w}}_0\), \({\varvec{w}}_1\), \({\varvec{w}}_2\) and \({\varvec{w}}_3\) correspond to the coefficients of \(\widetilde{m}\). As we will see in Remark 17, the system \({\varvec{w}} {\varvec{C}} = {\varvec{0}}\) has a solution \({\varvec{w}}^{\prime }\) whose 1\(\# \varLambda _X\)th entries equal zero. We choose such a solution as \({\varvec{w}}_3\). Assume the following condition:
Assumption 15
The entries in \({\varvec{s}}_1\) coincide with the \(\left( \# \varLambda _X + 1 \right) \)\(2 \# \varLambda _X\)th entries in \({\varvec{w}}_0 + {\varvec{w}}_3  {\varvec{z}}\), where \({\varvec{z}}\) is a closest lattice point in \(\mathcal{L}^{\prime }_3 := \langle {\varvec{w}}_1, {\varvec{w}}_2 \rangle _{\mathbb {Z}}\) to \({\varvec{w}}_0 + {\varvec{w}}_3\). In other words, \({\varvec{s}}_1\) is embedded in \({\varvec{w}}_0 + {\varvec{w}}_3  {\varvec{z}}\) as its \(\left( \# \varLambda _X + 1 \right) \)\(2 \# \varLambda _X\)th entries.
The lattice \(\mathcal{L}^{\prime }_3\) has rank 2, and thus we can expect to find \({\varvec{s}}_1\) in polynomial time by the Babai nearest plane algorithm [5] for solving CVP with sufficiently high probability under Assumption 15.
Remark 16
The reason why we assume Assumption 15 is the following: From the choice of \({\varvec{s}}_1\), the absolute values of the entries of \({\varvec{s}}_1\) are sufficiently smaller than those of \(\widetilde{{\varvec{m}}}\) and \({\varvec{r}}_1\). Thus we can expect that the value of \(\Vert {\varvec{w}}_0 + {\varvec{w}}_3  \left( a_1 {\varvec{w}}_1 + a_2 {\varvec{w}}_2 \right) \Vert \) is sufficiently small if certain entries of the vector \({\varvec{w}}_0 + {\varvec{w}}_3  \left( a_1 {\varvec{w}}_1 + a_2 {\varvec{w}}_2 \right) \) coincide with those of \({\varvec{s}}_1\).
Remark 17
Remark 18
4.2 Algorithm of our attack
Based on the idea in Sect. 4.1, we write down our attack algorithm against DEC in what follows. Let \((d, e, X(\underline{x}) ) \in \mathbb {Z}^2 \times \mathbb {Z}[ \underline{x} ]\) and \(( F_1(\underline{x}), F_2(\underline{x}), F_3(\underline{x}), N )\) \(\in ( \mathbb {Z}[ \underline{x} ] )^3 \times \mathbb {Z}\) be a public key and a ciphertext, as described in Sects 3.2 and 3.3. Let \(m(\underline{x}) \in \mathbb {Z}[ \underline{x} ]\) be a plaintext. Each cipher polynomial is of the form \(F_j (\underline{x}) = \widetilde{m}(\underline{x}) + s_j(\underline{x}) f(\underline{x}) + r_j(\underline{x}) X(\underline{x})\) for the twisted plaintext \(\widetilde{m}(\underline{x})\) and some random polynomials \(f (\underline{x})\), \(s_j (\underline{x})\) and \(r_j (\underline{x})\). We also recall that \(\varLambda _X\) and \(w_X\) denote the support of X and the total degree of X, respectively, see Notation in Sect. 1. Let \(\underline{{\varvec{k}}}\) be the maximal element of \(\varLambda _X\), see Remark 4 (2) for the ordering.

Input: \(\left( d, e, X (\underline{x} ) \right) \) and \(\left( F_1 (\underline{x}), F_2 (\underline{x}), F_3 (\underline{x}), N \right) \), a public key and a ciphertext.

Output: \(\widetilde{m} \left( \underline{x} \right) \), a twisted plaintext.
 Step 1: Determination of \(s_j^{\prime }:= s_j  s_{j+1}\) for \(j = 1\) and 2
 Step 11: Put \(F^{\prime }_j := F_j  F_{j+1}\), \(r^{\prime }_j := r_j  r_{j+ 1}\) (\(1 \le j \le 2\)) and put \(g := s^{\prime }_2r^{\prime }_1  s^{\prime }_1r^{\prime }_2\). Compute a basis of the kernel lattice of \({\varvec{A}}\), i.e., solve \({\varvec{u}} {\varvec{A}} = {\varvec{0}}\). This system is derived from unknown coefficients inwhere \({\varvec{A}}\) is the \(\left( 2 \# \varLambda _{X} + \# \varLambda _{X^2} \right) \times \# \varLambda _{X^3}\) coefficient matrix of the linear system obtained from the Eq. (6). Let \(\{ {\varvec{u}}^{\prime }_1, {\varvec{u}}^{\prime }_2, {\varvec{u}}^{\prime }_3 \}\) be the set of basis vectors for the kernel lattice.$$\begin{aligned} s^{\prime }_2 F^{\prime }_1  s^{\prime }_1F^{\prime }_2= & {} g X, \end{aligned}$$(6)

Step 12: We denote by \({\varvec{u}}_i\) the vector embedded in \({\varvec{u}}^{\prime }_i\) as its 1\(\left( 2 \# \varLambda _X \right) \)th entries for \(i = 1\), 2 and 3. Execute the weighted LLL reduction for the weight described in Sect. 4.1 to the lattice \(\mathcal{L}_1 := \langle {\varvec{u}}_1, {\varvec{u}}_2, {\varvec{u}}_3 \rangle \), and then get \(({\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2)\).

 Step 2: Obtaining a candidate of f
 Step 21: Compute a solution to \({\varvec{v}} {\varvec{B}} = {\varvec{b}}\). This system is derived from unknown coefficients inwhere \({\varvec{B}}\) is the \(\left( 3 \# \varLambda _{X} \times \# \varLambda _{X^2} \right) \) coefficient matrix obtained from the Eq. (7). Let \({\varvec{v}}_0\) be a solution, and let \(\{ {\varvec{v}}_1 \}\) be a basis of the kernel lattice \(\mathcal{L}_2\) of \({\varvec{B}}\). If \(\gcd (X, s^{\prime }_1) = 1\) in \(\mathbb {Z}[\underline{x}]\), then the lattice \(\mathcal{L}_2\) always has rank 1, see Remark 13.$$\begin{aligned} F^{\prime }_1 = s^{\prime }_1 f + r^{\prime }_1 X, \quad F^{\prime }_2 = s^{\prime }_2 f + r^{\prime }_2X, \end{aligned}$$(7)

Step 22: Compute \({\varvec{v}}^{\prime }_0 := {\varvec{v}}_0  \lfloor \langle {\varvec{v}}_0, {\varvec{v}}_1 \rangle / \langle {\varvec{v}}_1, {\varvec{v}}_1 \rangle \rceil {\varvec{v}}_1\), another solution to \({\varvec{v}} {\varvec{B}} = {\varvec{b}}\). Let \({\varvec{v}}_0^{\prime \prime }\) be the vector embedded in \({\varvec{v}}^{\prime }_0\) as its 1\((\# \varLambda _{X})\)th entries. Let \(f^{\prime } \left( \underline{x} \right) \in \mathbb {Z}[\underline{x}]\) be a polynomial with \({\varvec{f}^{\prime }} = {\varvec{v}}_0^{\prime \prime }\). Experimentally \({\varvec{v}}_0^{\prime }\) gives in many cases a polynomial closer to f than \({\varvec{v}}_0\), see Step 2 in Sect. 4.3.

 Step 3: Recovery of \(\widetilde{m}\)
 Step 31: Compute a solution to \({\varvec{w}} {\varvec{C}} = {\varvec{c}}\) and a basis of the kernel lattice of \({\varvec{C}}\). This system is derived from unknown coefficients inwhere \({\varvec{C}}\) is the \(( 3 \# \varLambda _{X} \times \# \varLambda _{X^2})\) coefficient matrix oftained from the Eq. (8) and \(f^{\prime }\) is the polynomial obtained in Step 22. Let \({\varvec{w}}_0\) be a solution and \(\{ {\varvec{w}}_1, {\varvec{w}}_2, {\varvec{w}}_3 \}\) a basis of the kernel lattice, denoted by \(\mathcal{L}_3\).$$\begin{aligned} F_1= & {} \widetilde{m} + s_1 f^{\prime } + r_1 X, \end{aligned}$$(8)

Step 32: Apply the Babai nearest plane algorithm to compute a closest lattice point \({\varvec{z}}\) in \(\mathcal{L}^{\prime }_3:= \langle {\varvec{w}}_1, {\varvec{w}}_2 \rangle _{\mathbb {Z}}\) to \({\varvec{w}}_0 + {\varvec{w}}_3\). Let \({\varvec{s}}_1\) be the vector embedded in \({\varvec{w}}_0 + {\varvec{w}}_3  {\varvec{z}}\) as its \(\left( \# \varLambda _X + 1 \right) \)\(2 \# \varLambda _X\)th entries.
 Step 33: Compute a solution to \({\varvec{x}} {\varvec{H}} = {\varvec{h}}\), the linear system derived from unknown coefficients inwhere \({\varvec{H}}\) is the \(\left( 2 \# \varLambda _X \times \# \varLambda _{X^2} \right) \) coefficient matrix obtained from the Eq. (9) and the coefficients of \(\widetilde{m}\) and r are indeterminates. Let \({\varvec{x}}\) be a solution to \({\varvec{x}} {\varvec{H}} = {\varvec{h}}\). Let \({\varvec{r}}^{\prime }\) be the vector consisting of the entries corresponding to r of \({\varvec{x}}\). Then we obtain a polynomial \(r^{\prime }\) whose coefficients coincide with those of r except the constant part, i.e., \(r = r^{\prime } + t\) for some \(t \in \mathbb {Z}\).$$\begin{aligned} F_1  \widetilde{m}  s_1 f^{\prime }= & {} r X, \end{aligned}$$(9)
 Step 34: ComputeOutput \(\widetilde{m} \left( \underline{x} \right) \).$$\begin{aligned} e^{\prime }:= & {} e^{1} \ (\mathrm{mod} \ \varphi \left( d \right) ), \\ H_1:= & {} F_1  s_1 f^{\prime }  r^{\prime } X, \\ \mu:= & {} c_{\underline{{\varvec{k}}}} \left( H_1 \right) , \\ c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right):= & {} \mu ^{e^{\prime }} \ \left( \mathrm{mod} \ d \right) \quad \left( 0< {c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right) }< d \right) , \\ c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right):= & {} \left( c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right) \right) ^{e} \ \left( \mathrm{mod} \ N d \right) \quad \left( 0< c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) < N d \right) , \\ t:= & {} \left( \mu  c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) \right) / c_{\underline{{\varvec{k}}}} \left( X \right) , \\ \widetilde{m}:= & {} F_1  s_1 f^{\prime }  \left( r^{\prime } + t \right) X. \end{aligned}$$

Remark 19
One may consider that applying the Babai nearest plane algorithm in terms of a weighted norm, or searching a desired vector \({\varvec{s}}_1\) by adding some elements in \(\mathcal{L}_3^{\prime }\) are effective. However, the oneway property of DEC can be broken with sufficiently high probability without such operations. We will see the details in Sect. 6. Hence in our attack let us omit these procedures.
Remark 20
In Step 34 of the above algorithm, we use the fact that \(c_{\underline{{\varvec{k}}}}(X)\) is divisible by d to compute an integer t, see (1) for the divisibility of \(c_{\underline{{\varvec{k}}}}(X)\).
4.3 Cryptanalysis of toy example
We break the oneway property of the instance in Sect. 3.6 of DEC. We use the same notations as in Sect. 3.6. In this case, we have \(\varLambda _g = \varLambda _{X^2} = \{ \left( 6, 0 \right) , \left( 3, 1 \right) , \left( 3, 0 \right) , \left( 0, 2 \right) , \left( 0, 1 \right) , \left( 0, 0 \right) \}\).
4.3.1 First step: determination of \(s^{\prime }_j = s_j  s_{j+1}\)
4.3.2 Second step: obtaining a candidate of f
4.3.3 Third Step: Recovery of \(\widetilde{m}\)
5 Complexity analysis
In this section, we investigate the complexity of the algorithm in Sect. 4.2. We analyse our attack in accordance with the parameter sizes in Sect. 3.5 (cf. Sect. 5 in [25]). Let \(X \in \mathbb {Z}[\underline{x}]\) be a public key of DEC. Let \(w_X\) and \(\varLambda _X\) denote the total degree and the support of X, respectively. To simplify the notations, we set \(w := w_X\), assume \(w = \# \varLambda _X\) and fix b, where b is the maximum of the bit length of the coefficients of X except its leading and constant terms. We show that the attack performs in polynomial time in terms of the parameters w and \(\lambda \). Here note that w and \(\lambda \) are independent of each other, see Remark 7 in Sect. 3. In our complexity analysis, we use the same notation as in Sect. 4.2. The parameters d and e are \(O \left( 2^\lambda \right) \) and \(O \left( w \lambda \right) \), respectively. Note that the size of each coefficient of \(F_j\) is \({O \left( w \lambda \right) }\) bits for \(j = 1\), 2 and 3, see Sect. 3.5 for the representation of the parameters by w and \(\lambda \). Assume that the size of each coefficient of \(s_1 f^{\prime }\) is bounded by \(O ( w \lambda )\) bits.
Remark 21
Remark 22
Second, we solve one or two linear systems in each step of our attack. Then, we obtain one solution and the kernel lattice for each linear system. We assume that the bit complexity of solving a nonhomogeneous linear system is equivalent to the bit complexity of computing the (row) Hermite Normal Form (HNF) of the augmented matrix of the system. According to Chapter 2 in [16], we assume that the computation of the HNF of an \(n \times m\) matrix \({\varvec{M}} = (M_{i,j})_{i,j}\) requires \(O(nm^4(\log (\Vert {\varvec{M}} \Vert _{\infty }))^2)\) bit operations, where \(\Vert {\varvec{M}} \Vert _{\infty } := \max _{i,j} \{ M_{i,j} \}\). On the other hand, we assume that a homogeneous linear system is solved by the Gaussian elimination.
To simplify the notations, we assume the sizes of the entries of one solution and an output basis of the kernel lattice of each linear system are \(O \left( \ell \right) \) bits if the sizes of the entries of its augmented matrix are \(O \left( \ell \right) \) bits.
Remark 23
Third, we discuss the size of the norm of a vector with integer entries. Let \({\varvec{a}} = \left( a_1, \ldots , a_k \right) \in \mathbb {Z}^k\) be a vector with \( a_i  \le 2^l\) for \(1 \le i \le k\). Since \(\Vert {\varvec{a}} \Vert \le \sqrt{ k 2^{2l} }\), the size of \(\Vert {\varvec{a}} \Vert \) is bounded by \(\mathrm{log} \left( \sqrt{ k 2^{2l} } \right) = \mathrm{log} \left( k^{1/2} \right) + l = O \left( \mathrm{log} \left( k \right) + l \right) \) bits. Similarly, the size of \(\Vert {\varvec{a}} \Vert ^2\) is \(O \left( \mathrm{log} \left( k \right) + l \right) \) bits.
5.1 The complexity of first step
5.2 The complexity of second step
5.3 The complexity of third step and the total complexity of our attack
Step 34 At the beginning of this step, we compute \(e':=e^{1} \ \mathrm{mod} \ \varphi \left( d \right) \) by using the extended Euclid’s algorithm. According to Remark 3.5 in [25], the integer d should be chosen so that one can compute \(\varphi \left( d \right) \) efficiently because the computation is needed in the decryption process (see [25], Sect. 3.4). In Remark 3.5 of [25], the integer d is expected to be a prime number as such an example. From this, we assume d is a prime number, and then we have \(\varphi \left( d \right) = d  1\).
Next, we compute \(c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right) := \mu ^{e^{\prime }} \ \left( \mathrm{mod} \ d \right) \) \(\left( 0< c_{\underline{{\varvec{k}}}} \left( m' \right) < d \right) \), where \(e^{\prime } := e^{1}\) (mod \(\varphi (d)\)) and \({\mu }\) is a certain coefficient of \(H_1 \left( \underline{x} \right) \) (cf. Step 34 in Sect. 4.2). Recall that the bit sizes of \(e^{\prime }\), \(\mu \) and d are \(O \left( \lambda \right) \), \(O \left( w \lambda \right) \) and \(O \left( \lambda \right) \), respectively. Thus this computation can be done in \(O \left( w \lambda ^2 + \lambda ^3 \right) \) bit operations by the squareandmultiply algorithm for modular exponentiation.
Theorem 24
Consequently, our attack performs in polynomial time for all the parameters \(\lambda \) and \(w_X\), where \(\lambda \) and \(w_X\) are independent of each other.
Remark 25
The estimated complexity in Theorem 24 shows that the computation of our attack may become expensive for large \(w=w_X\) and \(\# \varLambda _X \le w\). Thus, to secure DEC, one can think of increasing the parameters w and \(\# \varLambda _X\). However, DEC is impractical for large \(w_X\) and \(\# \varLambda _X\) since ciphertexts of DEC have exceedingly large sizes. For example, when \(w_X = \# \varLambda _X = 45\), \(b = 10\) and \(\lambda = 128\), we generated 100 ciphertexts \((F_1, F_2, F_3, N)\) in accordance with Sects. 3.2 and 3.3, and measured their sizes. As a result, their average size is about 10, 086, 237 bits.
Remark 26
From the above reason, the dominant term of the estimated complexity in Theorem 24 is \(O (w^4 \lambda ^3)\) in practice.
6 Experimental verification
In this section, we demonstrate with experimental results that our attack algorithm enables one to break the oneway property of DEC in practical time. In our experiments, we generated DEC instances of \(n=4\), where n is the number of indeterminates of a public key \(X(\underline{x})\). The PC used in our experiments is as follows: The OS is Mac OS X, 64 bit. The processor is 2.60GHz CPU (Intel Corei5). The memory is 16GB. Authors implemented the attack algorithm over Magma V2.213 [8]. For the parameters, we adopted recommended ones in Remark 9 (such parameters shall make DEC instances \(\lambda =128\) bit level secure).
Table 1 indicates results of our experiments on our cryptanalysis of DEC instances. In Step 1 of the table, the number of successes is shown only if the target lattice point \(\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right) \) or \( \left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right) \) is found. For the target lattice point, see Step 1 of Attack Algorithm in Sect. 4.2. In Step 3 of the table, the number of successes is shown only if we succeeded in finding a twisted plaintext \(\widetilde{m}\) (or \( \widetilde{m}\)).
We see from the results of Step 1 in Table 1 that the weighted LLL reduction recovered our target lattice point in Step 1 with high probability, being about from 70 to 90%. On the other hand, we could not find the target lattice point with the usual LLL reduction in any case of our experiments (we omit to show the experimental results on the attack with the usual LLL reduction). We see from the results of Step 3 in Table 1 that our attack algorithm with the weighted LLL reduction could find the twisted plaintext \(\widetilde{m}\) (or \( \widetilde{m}\)) with sufficiently high probability, being about from 20 to 40%. We, however, could not succeed in finding the twisted plaintext at all by another one with the usual LLL reduction. From this, we infer that to adopt the weighted LLL reduction is quit important for our attack to succeed, and that our attack with the weighted LLL reduction has sufficiently high success probability for practical cryptanalysis.
Experimental results on Attack Algorithm given in Sect. 4.2 for DEC of 128 bit level security with 4 indeterminates
Parameters recommended in Sect. 3.5  Experimental results  

Value of b  Number of monomials of X  Number of successes of (1st and 3rd Steps in) the attack / 100  
1st Step  3rd Step  Ave. Time (s)  
10  3  71  23  0.02 
4  81  33  0.03  
5  86  29  0.04  
6  86  35  0.06  
7  85  29  0.07  
8  92  33  0.09  
9  88  41  0.21  
10  91  32  0.25  
50  3  68  21  0.02 
4  82  39  0.03  
5  77  30  0.04  
6  83  33  0.07  
7  88  41  0.08  
8  93  32  0.10  
9  92  36  0.21  
10  91  34  0.28  
100  3  75  29  0.02 
4  78  26  0.03  
5  80  36  0.04  
6  83  31  0.07  
7  82  34  0.08  
8  95  40  0.11  
9  87  36  0.20  
10  91  38  0.27 
Remark 27
The ranks of lattices occurring in Step 1 are equal to 3 in many cases. In fact, this is true for 100 instances of DEC constructed in our experiments. The LLL reduction finds shortest vectors in such lattices of low rank with high probability. In Step 1, a weighted norm is determined so that the target vector becomes a (nearly) shortest vector in terms of the norm. Thus the most important vector for our attack (the target vector in Step 1) is found by the weighted LLL reduction with high probability.
Remark 28
The existence of some failures of our attack suggests that there may exist a method to resist our attack. We analyzed some failure cases and found a reason why our attack failed in finding target lattice points in Steps 1 and 3. In Step 1 of each failure case, the weighted LLL algorithm found a shortest vector, but our target lattice point was not shortest. Similarly, in Step 3 of each failure case, our target lattice point was not a closest vector, while the Babai nearest plane algorithm found a closest vector. Therefore one may resist our attack if it is possible to choose random polynomials or public/secret keys such that our target lattice points are not shortest or closest in lattices ocurring in Steps 1 and 3. However, special choices of polynomials may lead us to another attack, and adding brute force methods to our attack seems to find target lattices points in such cases (see below). In order to resist our attack, we conclude that a major improvement of DEC is required. For example, the number of ciphertexts (polynomials) should be reduced from 3 to 2 or 1 because using 3 ciphertexts is essential to our attack.
On the other hand, we consider whether there is room for improving our attack or not. A simple improvement is to add steps of brute force search (with small range) to Steps 1 and 3. Our analysis in Sect. 4.1 suggests that our target vectors in Steps 1 and 3 are nearly shortest and nearly closest vectors, respectively, and thus our target vectors seem to be found by brute force methods with small range. However, we omit to conduct experiments on our attack with brute force methods. As mentioned above, we believe that our attack has already provided a practical solution to a problem of breaking DEC which is a candidate of PQC with small key sizes.
7 Conclusion
We present in this paper a polynomial timeattack based on the weighted LLL reduction against the oneway property of a Diophantine Equationbased Cryptosystem (DEC), which was proposed in 2015 by the third author of this paper as one of the candidates of PostQuantum Cryptosystems (PQC). Compared with other wellknown candidates of PQC, sizes of public keys in DEC are much smaller, e.g., about 1, 200 bits for 128 bit level security. This is a strongly desired characteristic for candidates of PQC.
Diophantine equations are generally unsolvable, and thus it is expected to be a base of the security of PQC. However, we showed that DEC’s security does not rely on the computational hardness to solve Diophantine equations, and that moreover DEC is no longer secure. Concretely, with linearlization technique, one can reduce breaking the oneway property of DEC to computing certain (comparatively) shorter points in low ranklattices. Our most crucial target lattice point has the following special property: it is not necessarily a shortest lattice point whereas most of the entries are comparatively small. In our attack, even with the LLL reduction in terms of wellknown norms, e.g., pnorms for \(1 \le p \le \infty \), one seems to fail in finding such lattice points.
The most (heuristically)technical point in our attack is changing the norm in the LLL reduction from the Euclidean norm to an appropriate weighted one. One can see from our analysis that the most important target lattice point becomes a (nearly) shortest lattice point in terms of a weighted norm, where the weight is determined by our heuristic method. Furthermore, the most important target lattice point is embedded in a (weighted) lattice of 3rank, which implies the weighted LLL reduction can output with high probability such a target point. From this, we applied the weighted LLL reduction, which is the LLL reduction in terms of a weighted norm to our cryptanalysis. Our experimental results and complexity analysis suggest that for all the recommended parameters, the oneway property of DEC can be broken with sufficiently high probability by our polynomial timeattack based on the weighted LLL reduction.
We also demonstrated with our experimental results that the weighted LLL reduction gives an effective computational tool to find lattice points of special characteristic: the sizes of entries are almost known and most of them are small. Hence the weighted LLL reduction can provide a tool to investigate the security of cryptosystems whose security are transformed to the problem of computing such lattice points.
Notes
Acknowledgements
The authors deeply thank Shun’ichi Yokoyama for many helpful comments, corrections, suggestions on this research, and discussions in the implementations on Magma. The authors also thank Steven Galbraith for helpful comments on Coppersmith’s method, and thank Masaya Yasuda for helpful comments on the weighted LLL reduction and corrections on this paper. This work was supported by JST CREST Grant Number JPMJCR14D6, Japan. The authors are grateful to the anonymous referees for their careful reading of our manuscript and their valuable comments and suggestions.
References
 1.Akiyama, K., Goto, Y.: An algebraic surface publickey cryptosystem. IEICE Tech. Rep. 104(421), 13–20 (2004)Google Scholar
 2.Akiyama, K., Goto, Y.: A Publickey Cryptosystem using Algebraic Surfaces, In: Proceedings of PQCrypto., pp. 119–138, (2006). http://postquantum.cr.yp.to/. Accessed 19 June 2018
 3.Akiyama, K., Goto, Y.: An improvement of the algebraic surface publickey cryptosystem. In: Proceedings of 2008 Symposium on Cryptography and Information Security, SCIS 2008, CDROM, 1F12, (2008)Google Scholar
 4.Akiyama, K., Goto, Y., Miyake, H.: An algebraic surface cryptosystem. In: Jarecki, S., Tsudik, G. (eds.) Public Key Cryptography – PKC 2009. PKC 2009. Lecture Notes in Computer Science, vol. 5443. Springer, Berlin, Heidelberg (2009)CrossRefGoogle Scholar
 5.Babai, L.: On Lovász’ lattice reduction and the nearest lattice point problem. Combinatorica 6(1), 1–13 (1986). (Preliminary version in STACS 1985)MathSciNetCrossRefGoogle Scholar
 6.Bérczes, A., Hajdu, L., HirataKohno, N., Kovács, T., Pethö, A.: A key exchange protocol based on Diophantine equations and Sintegers. JSIAM Lett. 6, 85–88 (2014)MathSciNetCrossRefGoogle Scholar
 7.Bernstein, D.J., Buchmann, J., Dahmen, E. (eds.): PostQuantum Cryptography. SpringerVerlag, Berlin (2009)zbMATHGoogle Scholar
 8.Bosma, W., Cannon, J., Playoust, C.: The Magma algebra system. I. The user language. J. Symb. Comput. 24(3–4), 235–265 (1997)MathSciNetCrossRefGoogle Scholar
 9.Coppersmith, D.: Small solutions to polynomial equations, and low exponent RSA vulnerabilities. J. Cryptol. 10(4), 233–260 (1997). SpringerVerlagMathSciNetCrossRefGoogle Scholar
 10.Cusick, T.W.: Cryptoanalysis of a public key system based on diophantine equations. Inf. Process. Lett. 56(2), 73–75 (1995)CrossRefGoogle Scholar
 11.Ding, J., Gower, J.E., Schmidt, D.S.: Multivariate public key cryptosystems, advances in information security, 25. Springer, Berlin, Heidelberg (2006)zbMATHGoogle Scholar
 12.Davis, M., Matijasevič, Y., Robinson, J.: Hilbert’s tenth problem, Diophantine equations: positive aspects of a negative solution, mathematical developments arising from Hilbert problems In: Browder, F.E. (ed.) Proceedings of Symposia in Pure Mathematics, vol. 28, pp. 1–34. American Mathematical Society, Providence (1976)Google Scholar
 13.Eisenträger, K.: Hilbert’s tenth problem for function fields of varieties over number fields and padic fields. J. Algebra 310(2), 775–792 (2007)MathSciNetCrossRefGoogle Scholar
 14.Faugère, J. C., Goyet, C., Renault, G.: Attacking (EC)DSA given only an implicit hint. In: Knudsen, L.R., Wu, H. (eds.) Selected Areas in Cryptography. SAC 2012. Lecture Notes in Computer Science, vol. 7707. Springer, Berlin, Heidelberg (2012)Google Scholar
 15.Faugère, J. C., Spaenlehauer, P. J.: Algebraic cryptanalysis of the PKC’2009 algebraic surface cryptosystem. In: Nguyen, P.Q., Pointcheval, D. (eds.) Public Key Cryptography – PKC 2010. PKC 2010. Lecture Notes in Computer Science, vol. 6056. Springer, Berlin, Heidelberg (2010)CrossRefGoogle Scholar
 16.Galbraith, S.D.: Mathematics of public key cryptography. Cambridge University Press, New York (2012)CrossRefGoogle Scholar
 17.HirataKohno, N., Pethö, A.: On a key exchange protocol based on Diophantine equations. Infocommunications J. 5(3), 17–21 (2013). (Scientific Association for Infocommunications (HTE)) Google Scholar
 18.Iwami, M.: A reduction attack on algebraic surface publickey cryptosystems, lecture notes in computer science, vol. 5081, pp. 323–332. Springer, Berlin (2008)zbMATHGoogle Scholar
 19.Lenstra, A.K., Lenstra, H.W., Lovász, L.: Factoring polynomials with rational coefficients. Math. Ann. 261(4), 515–534 (1982). (SpringerVerlag) MathSciNetCrossRefGoogle Scholar
 20.Lin, C.H., Chang, C.C., Lee, R.C.T.: A new publickey cipher system based upon the diophantine equations. IEEE Trans. Comput. 44(1), 13–19 (1995). IEEE Computer Society Washington, DC, USACrossRefGoogle Scholar
 21.Lyubashevsky, V., Peikert, C., Regev, O.: On ideal lattices and learning with errors over rings. In: Gilbert, H. (ed.) Advances in Cryptology – EUROCRYPT 2010. EUROCRYPT 2010. Lecture Notes in Computer Science, vol. 6110. Springer, Berlin, Heidelberg (2010)Google Scholar
 22.Misoczki, R., Tillich, J.P., Sendrier, N., Barreto, P.S.L.M.: MDPCMcEliece: New McEliece Variants from Moderate Density ParityCheck Codes, Information Theory Proceedings. (ISIT), IEEE International Symposium on Information Theory (2013)Google Scholar
 23.A draft of the report on postquantum cryptography NISTIR 8105. http://csrc.nist.gov/publications/drafts/nistir8105/nistir_8105_draft.pdf. Accessed 19 June 2018
 24.The slides of NIST’s announcement “PostQuantum Cryptography: NIST’s Plan for the Future”. https://pqcrypto2016.jp/data/pqc2016_nist_announcement.pdf. Accessed 19 June 2018
 25.Okumura, S.: A public key cryptosystem based on diophantine equations of degree increasing type. Pac. J. Math. Ind. 7(4), 33–45 (2015). (Springer, Berlin Heidelberg) MathSciNetzbMATHGoogle Scholar
 26.Pheidas, T.: Hilbert’s tenth problem for fields of rational functions over finite fields. Invent. Math. 103(1), 1–8 (1991). (SpringerVerlag) MathSciNetCrossRefGoogle Scholar
 27.Tao, C., Diene, A., Tang, S., Ding, J.: Simple matrix scheme for encryption. In: Gaborit, P. (ed.) PostQuantum Cryptography. PQCrypto 2013. Lecture Notes in Computer Science, vol. 7932. Springer, Berlin, Heidelberg (2013)CrossRefGoogle Scholar
 28.Uchiyama, S., Tokunaga, H.: On the Security of the Algebraic Surface Publickey Cryptosystems (in Japanese), In: Proceedings of 2007 Symposium on Cryptography and Information Security, SCIS 2007, CDROM, 2C12, (2007)Google Scholar
 29.Videla, C.R.: Hilbert’s tenth problem for rational function fields in characteristic 2. Proc. Am. Math. Soc. 120(1A), 249–253 (1994). (American Mathematical Society) MathSciNetzbMATHGoogle Scholar
 30.Voloch, F.: Breaking the akiyamagoto cryptosystem, contemporary mathematics. Arith. Geom. Cryptogr. Coding Theory 487, 113–118 (2007). (American Mathematical Society, Providence, RI) zbMATHGoogle Scholar
 31.Yosh, H.: The key exchange cryptosystem used with higher order diophantine equations. Int. J. Netw. Secur. Appl. J. 3(2), 43–50 (2011)Google Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.