General Linear Group Action on Tensors: A Candidate for Post-quantum Cryptography

Abstract

Starting from the one-way group action framework of Brassard and Yung (Crypto’90), we revisit building cryptography based on group actions. Several previous candidates for one-way group actions no longer stand, due to progress both on classical algorithms (e.g., graph isomorphism) and quantum algorithms (e.g., discrete logarithm).

We propose the general linear group action on tensors as a new candidate to build cryptography based on group actions. Recent works (Futorny–Grochow–Sergeichuk Lin. Alg. Appl., 2019) suggest that the underlying algorithmic problem, the tensor isomorphism problem, is the hardest one among several isomorphism testing problems arising from areas including coding theory, computational group theory, and multivariate cryptography. We present evidence to justify the viability of this proposal from comprehensive study of the state-of-art heuristic algorithms, theoretical algorithms, hardness results, as well as quantum algorithms.

We then introduce a new notion called pseudorandom group actions to further develop group-action based cryptography. Briefly speaking, given a group G acting on a set S, we assume that it is hard to distinguish two distributions of (s, t) either uniformly chosen from $S\times S$, or where s is randomly chosen from S and t is the result of applying a random group action of $g\in G$ on s. This subsumes the classical Decisional Diffie-Hellman assumption when specialized to a particular group action. We carefully analyze various attack strategies that support instantiating this assumption by the general linear group action on tensors.

Finally, we construct several cryptographic primitives such as digital signatures and pseudorandom functions. We give quantum security proofs based on the one-way group action assumption and the pseudorandom group action assumption.

Full version of this paper is available at ia.cr/2019/687.

1 Introduction

Modern cryptography has thrived thanks to the paradigm shift to a formal approach: precise definition of security and mathematically sound proof of security of a given construction based on accurate assumptions. Most notably, computational assumptions originated from specific algebraic problem such as factoring and discrete logarithm have enabled widely deployed cryptosystems.

Clearly, it is imperative to base cryptography on diverse problems to reduce the risk that some problems turn out to be easy. One such effort was by Brassard and Yung soon after the early development of modern cryptography [17]. They proposed an approach to use a group action to construct a one-way function, from which they constructed cryptographic primitives such as bit commitment, identification and digital signature. The abstraction of one-way group actions ($\mathrm {OWA}$) not only unifies the assumptions from factoring and discrete logarithm, but more importantly Brassard and Yung suggested new problems to instantiate it such as the graph isomorphism problem (GI). Since then, many developments fall in this framework [26, 46, 65, 70]. In particular, the work of Couveignes [26] can be understood as a specific group action based on isogenies between elliptic curves, and it has spurred the development of isogeny-based cryptography [28].

However, searching for concrete group actions to support this approach turns out to be a tricky task, especially given the potential threats from attackers capable of quantum computation. For graph isomorphism, there are effective heuristic solvers [63, 64] as well as efficient average-case algorithms [7], not to mention Babai’s recent breakthrough of a quasipolynomial-time algorithm [5]. Shor’s celebrated work solves discrete logarithm and factoring in polynomial time on a quantum computer [79], which would break a vast majority of public-key cryptography. The core technique, quantum Fourier sampling, has proven powerful and can be applied to break popular symmetric-key cryptosystems as well [54]. A subexponential-time quantum algorithm was also found for computing isogenies in ordinary curves [23], which attributes to the shift to super-singular curves in the recent development of isogeny-based cryptography [40]. In fact, there is a considerable effort developing post-quantum cryptography that can resist quantum attacks. Besides isogeny-based, there are popular proposals based on discrete lattices, coding problems, and multivariate equations [10, 22].

1.1 Overview of Our Results

In this paper, we revisit building cryptography via the framework of group actions and aim to provide new candidate and tools that could serve as quantum-safe solutions. Our contribution can be summarized below.

First, we propose a family of group actions on tensors of order at least three over a finite field as a new candidate for one-way actions. We back up its viability by comparison with other group actions, extensive analysis from heuristic algorithms, provable algorithmic and hardness results, as well as demonstrating its resistance to a standard quantum Fourier sampling technique.

Second, we propose the notion of pseudorandom group actions ($\mathrm {PRA}$) that extends the scope of the existing group-action framework. The $\mathrm {PRA}$ assumption can be seen as a natural generalization of the Decisional Diffie-Hellman (DDH) assumption. We again instantiate it with the group action on tensors, and we provide further evidence (in addition to those for one-wayness) by analyzing various state-of-art attacking strategies.

Finally, based on any $\mathrm {PRA}$, we show realization of several primitives in Minicrypt such as digital signatures via the Fiat-Shamir transformation and pseudorandom functions. We give complete security proofs against quantum adversaries, thanks to recent advances in analyzing quantum superposition attacks and the quantum random oracle model [81, 82, 85], which is known to be a tricky business. Our constructions based on $\mathrm {PRA}$ are more efficient than known schemes based on one-way group actions. As a side contribution, we also describe formal quantum-security proofs for several $\mathrm {OWA}$-based schemes including identification and signatures, which are incomplete in the literature and deserve some care.

In what follows, we elaborate on our proposed group action based on tensors and the new pseudorandom group action assumption. Readers interested in the cryptographic primitives supported by $\mathrm {PRA}$ are referred to the full version of this paper [53].

The General Linear Group Action on Tensors. The candidate group action we propose is based on tensors, a central notion in quantum theory. In this paper, a k-tensor T is a multidimensional array with k indices $i_1, i_2, \ldots , i_k$ over a field $\mathbb {F}$, where $i_j \in \{1, 2, \ldots , d_j\}$ for $j=1, 2, \ldots , k$. For a tuple of indices $(i_1, i_2, \ldots , i_k)$, the corresponding component of T denoted as $T_{i_1, i_2, \ldots , i_k}$ is an element of $\mathbb {F}$. The number k is called the order of the tensor. A matrix over field $\mathbb {F}$ can be regarded as a tensor of order two.

We consider a natural group action on k-tensors that represents a local change of basis. Let $G = \prod _{j=1}^k {{\,\mathrm{\mathrm {GL}}\,}}(d_j,\mathbb {F})$ be the direct product of general linear groups. For $M = \bigl ( M^{(j)} \bigr )_{j=1}^k \in G$, and a k-tensor T, the action of M on T is given by

$$\begin{aligned} \alpha : (M, T) \mapsto \widehat{T}, \text { where } \widehat{T}_{i_1, i_2, \ldots , i_k} = \sum _{l_1, l_2, \ldots , l_k} \biggl ( \prod _{j=1}^k M^{(j)}_{i_j,l_j} \biggr ) T_{l_1, l_2, \ldots , l_k}. \end{aligned}$$

We shall refer to the above group action as the general linear group action on tensors ($\mathrm {GLAT}$) of dimensions $(d_1, \dots , d_k)$ over $\mathbb {F}$, or simply $\mathrm {GLAT}$ when there is no risk of confusion. We will consider group actions on tensors of order at least three, as the problem is usually easy for matrices. In fact, in most of the cases, we focus on 3-tensors which is most studied and believed to be hard.

General Linear Actions on Tensors as a Candidate for One-Way Group Actions. We propose to use $\mathrm {GLAT}$ as an instantiation of one-way group actions. Roughly speaking, a group action is called a one-way group action (OWA in short), if for a random $s\in S$, a random $g\in G$, $t=g\cdot s$, and any polynomial-time adversary $\mathcal {A}$ given s and t as input, $\mathcal {A}$ outputs a $g'\in G$ such that $t=g'\cdot s$ only with negligible probability.

Breaking the one-wayness can be identified with solving some isomorphism problem. Specifically, two k-tensors T and $\widehat{T}$ are said to be isomorphic if there exists an $M\in G$ such that $\widehat{T} = \alpha (M,T)$. We define the decisional tensor isomorphism problem (DTI) as deciding if two given k-tensors are isomorphic; and the search version (TI) is tasked with computing an $M\in G$ such that $\widehat{T} = \alpha (M, T)$ if there is one. Clearly, our assumption that $\mathrm {GLAT}$ is a one-way group action is equivalent to assuming that TI is hard for random $M\in G$, random k-tensor S, and $T := \alpha (M,S)$. We focus on the case when the order k of the tensor equals three and the corresponding tensor isomorphism problem is abbreviated as 3TI. We justify our proposal from multiple routes; see Sect. 3 for a more formal treatment.

• 1. The 3-tensor isomorphism problem can be regarded as “the most difficult” one among problems about testing isomorphism between objects, such as polynomials, graphs, linear codes, and groups, thanks to the recent work of Futorny, Grochow, and Sergeichuk [39]. More specifically, it was proven in [39] that several isomorphism problems, including graph isomorphism, quadratic polynomials with 2 secrets from multivariate cryptography [70], p-group isomorphism from computational group theory [59, 69], and linear code permutation equivalence from coding theory [73, 77], all reduce to 3TI; cf. Observation 2. Note that testing isomorphism of quadratic polynomials with two secrets has been studied in multivariate cryptography for more than two decades [70]. Isomorphism testing of p-groups has been studied in computational group theory and theoretical computer science at least since the 1980’s (cf. [59, 69]). Current status of these two problems then could serve as evidence for the difficulty of 3TI.

• 2. Known techniques that are effective on GI, including the combinatorial techniques [83] and the group-theoretic techniques [3, 60], are difficult to translate to 3TI. Indeed, it is not even clear how to adapt a basic combinatorial technique for GI, namely individualizing a vertex [7], to the 3TI setting. It is also much harder to work with matrix groups over finite fields than to work with permutation groups. Also, techniques in computer algebra, including those that lead to the recent solution of isomorphism of quadratic polynomials with one secret [50], seem not applicable to 3TI.

• 3. Finally, there is negative evidence that quantum algorithmic techniques involving the most successful quantum Fourier sampling may not be able to solve GI and code equivalence [34, 45]. It is expected that the same argument holds with respect to 3TI as well. Loosely speaking, this is because the group underlying 3TI is a direct product of general linear groups, which also has irreducible representations of high dimensions.

A New Assumption: Pseudorandom Group Actions. Inspired by the Decisional Diffie-Hellman assumption, which enables versatile cryptographic constructions, we propose the notion of pseudorandom group actions, or $\mathrm {PRA}$ in short.

Roughly speaking, we call a group action $\alpha : G \times S \rightarrow S$ pseudorandom, if any quantum polynomial-time algorithm $\mathcal {A}$ cannot distinguish the following two distributions except with negligible probability: (s, t) where $s, t\in _R S$, and the other distribution $(s, \alpha (g, s))$, where $s\in _R S$ and $g\in _R G$. A precise definition can be found in Sect. 4.

Note that if a group action is transitive, then the pseudorandom distribution trivially coincides with the random distribution. Unless otherwise stated, we will consider intransitive group actions when working with pseudorandom group actions. In fact, we can assume that (s, t) from the random distribution are in different orbits with high probability, while (s, t) from the pseudorandom distribution are always in the same orbit.

Also note that $\mathrm {PRA}$ is a stronger assumption than $\mathrm {OWA}$. To break $\mathrm {PRA}$, it is enough to solve the isomorphism testing problem on average in a relaxed sense, i.e., on $1/{{\,\mathrm{\mathrm {poly}}\,}}(n)$ fraction of the input instances instead of all but $1/{{\,\mathrm{\mathrm {poly}}\,}}(n)$ fraction, where n is the input size.

The Decisional Diffie-Hellman (DDH) assumption [13, 32] can be seen as the $\mathrm {PRA}$ initiated with a certain group action; see Observation 4. However, DDH is broken on a quantum computer. We resort again to $\mathrm {GLAT}$ as a quantum-safe candidate of $\mathrm {PRA}$. We investigate the hardness of breaking $\mathrm {PRA}$ from various perspectives and provide further justification for using the general linear action on 3-tensors as a candidate for $\mathrm {PRA}$.

• 1. Easy instances on 3-tensors seem scarce, and average-case algorithms do not speed up dramatically. Indeed, the best known average-case algorithm, while improves over worst-case somewhat due to the birthday paradox, still inherently enumerate all vectors in $\mathbb {F}_q^n$ and hence take exponential time [15, 59].

• 2. For 3-tensors, there have not been non-trivial and easy-to-compute isomorphism invariants, i.e., those properties that are preserved under the action. For example, a natural isomorphism invariant, the tensor rank, is well-known to be NP-hard [47]. Later work suggests that “most tensor problems are NP-hard” [49].

• 3. We propose and analyze several attack strategies from group theory and geometry. While effective on some non-trivial actions, these attacks do not work for the general linear action on 3-tensors. For instance, we notice that breaking our $\mathrm {PRA}$ from $\mathrm {GLAT}$ reduces to the orbit closure intersection problem, which has received considerable attention in optimization, and geometric complexity theory. Despite recent advances [1, 19, 20, 29, 52, 66], any improvement towards a more effective attack would be a breakthrough.

Recently, De Feo and Galbraith proposed an assumption in the setting of supersingular isogeny-based cryptography, which can be viewed as another instantiation of $\mathrm {PRA}$  [36, Problem 4]. This gives more reason to further explore $\mathrm {PRA}$ as a basic building block in cryptography.

1.2 Discussions

In this paper, we further develop and extend the scope of group action based cryptography by introducing the general linear group actions on tensors ($\mathrm {GLAT}$) and formulating the pseudorandom assumption, generalizing the DDH assumption. We construct and prove the quantum security of various cryptographic primitives such as signatures and pseudorandom functions in this framework.

There are two key features of $\mathrm {GLAT}$ that are worth mentioning explicitly. First, the general linear action is non-commutative simply because the general linear group is non-abelian. This is, on the one hand, an attractive property that enabled us to argue the quantum hardness and the infeasibility of quantum Fourier sampling type of attacks. On the other hand, however, this also makes it challenging to extend many attractive properties of discrete-logarithm and decisional Diffie-Hellman to the more general framework of group action cryptography. For example, while it is known that the worst-case DDH assumption reduces to the average-case DDH assumption [68], the proof relies critically on commutativity. Second, the general linear action is linear and the space of tensors form a linear space. Linearity seems to be responsible for the supergroup attacks on the $\mathrm {PRA} (d)$ assumption discussed in Sect. 5.1. It also introduces the difficulty for building more efficient PRF constructions analogous to the DDH-based ones proposed in [68].

Our work leaves a host of basic problems about group action based cryptography as future work. First, we have been focusing on the general linear group actions on tensors. A mixture of different types of group actions on different indices of the tensor may enable more efficient constructions or other appealing structural properties. It will be interesting to investigate how the hardness varies with the group actions on tensors, and identify group actions for practicability considerations. Second, it is appealing to recover the average-case to worst-case reduction, at least to some extent, for the general group actions framework. Finally, it is an important open problem to build quantum-secure public-key encryption schemes based on hard problems about $\mathrm {GLAT}$ or its close variations.

2 The Group Action Framework

In this section, we formally describe the framework for group action based cryptography to be used in this paper. While such general frameworks were already proposed by Brassard and Yung [17] and Couveignes [26], there are delicate differences in several places, so we will have to still go through the details. This section should be considered as largely expository.

2.1 Group Actions and Notations

Let us first formally define group actions. Let G be a group, S be a set, and $\mathrm {id}$ the identity element of G. A (left) group action of G on S is a function $\alpha : G\times S\rightarrow S$ satisfying the following: (1) $\forall s\in S$, $\alpha (\mathrm {id}, s)=s$; (2) $\forall g, h\in G$, $s\in S$, $\alpha (gh, s)=\alpha (g, \alpha (h, s))$. The group operation is denoted by $\circ$, e.g. for $g, h\in G$, we can write their product as $g\circ h$. We shall use $\cdot$ to denote the left action, e.g. $g\cdot s=\alpha (g, s)$. We may also consider the right group action $\beta :S\times G\rightarrow S$, and use the exponent notation for right actions, e.g. $s^g=\beta (s, g)$.

Later, we will use a special symbol $\bot \not \in G\cup S$ to indicate that a bit string does not correspond to an encoding of an element in G or S. We extend the operators $\circ$ and $\cdot$ to $\circ :G\cup \{\bot \}\times G\cup \{\bot \}\rightarrow G\cup \{\bot \}$ and $\cdot :G\cup \{\bot \}\times S\cup \{\bot \}\rightarrow S\cup \{\bot \}$, by letting $g\circ h=\bot$ whenever $g=\bot$ or $h=\bot$, and $g\cdot s=\bot$ whenever $g=\bot$ or $s=\bot$.

Let $\alpha :G\times S\rightarrow S$ be a group action. For $s\in S$, the orbit of s is $O_s=\{t\in S : \exists g\in G, g\cdot s=t\}$. The action $\alpha$ partitions S into a disjoint union of orbits. If there is only one orbit, then $\alpha$ is called transitive. Restricting $\alpha$ to any orbit O gives a transitive action. In this case, take any $s\in O$, and let $\mathrm {Stab}(s, G)=\{g\in G : g\cdot s=s\}$ be the stabilizer group of s in G. For any $t\in O$, those group elements sending s to t form a coset of $\mathrm {Stab}(s, G)$. We then obtain the following easy observation.

Observation 1

Let $\alpha :G\times S\rightarrow S$, s, and O be as above. The following two distributions are the same: the uniform distribution of $t\in O$, and the distribution of $g\cdot s$ where g is sampled from a uniform distribution over G.

2.2 The Computational Model

For computational purposes, we need to model the algorithmic representations of groups and sets, as well as basic operations like group multiplication, group inverse, and group actions. We review the group action framework as proposed in Brassard and Yung [17]. A variant of this framework, with a focus on restricting to abelian (commutative) groups, was studied by Couveignes [26]. However, it seems to us that some subtleties are present, so we will propose another version, and compare it with those by Brassard and Yung, and Couveignes, later.

• Let n be a parameter which controls the instance size. Therefore, polynomial time or length in the following are with respect to n.

• (Representing group and set elements.) Let G be a group, and S be a set. Let $\alpha : G\times S\rightarrow S$ be a group action. Group elements and set elements are represented by bit strings $\{0, 1\}^*$. There are polynomials p(n) and q(n), such that we only work with group elements representable by $\{0, 1\}^{p(n)}$ and set elements representable by $\{0, 1\}^{q(n)}$. There are functions $F_G$ and $F_S$ from $\{0, 1\}^*$ to $G\cup \{\perp \}$ and $S\cup \{\perp \}$, respectively. Here, $\perp$ is a special symbol, designating that the bit string does not represent a group or set element. $F_G$ and $F_S$ should be thought of as assigning bit strings to group elements.

• (Unique encoding of group and set elements.) For any $g\in G$, there exists a unique $b\in \{0, 1\}^*$ such that $F_G(b)=g$. In particular, there exists a unique bit string, also denoted by $\mathrm {id}$, such that $F_G(\mathrm {id})=\mathrm {id}$. Similarly, for any $s\in S$, there exists a unique $b\in \{0, 1\}^*$ such that $F_S(b)=s$.

• (Group operations.) There are polynomial-time computable functions $\mathrm {PROD}:\{0, 1\}^*\times \{0, 1\}^*\rightarrow \{0, 1\}^*$ and $\mathrm {INV}:\{0, 1\}^*\rightarrow \{0, 1\}^*$, such that for $b, c\in \{0, 1\}^*$, $F_G(\mathrm {PROD} (b, c))=F_G(b)\circ F_G(c)$, and $F_G(\mathrm {INV} (b))\circ F_G(b)=\mathrm {id}$.

• (Group action.) There is a polynomial-time function $a:\{0, 1\}^*\times \{0, 1\}^*\rightarrow \{0, 1\}^*$, such that for $b\in \{0, 1\}^*$ and $c\in \{0, 1\}^*$, satisfies $F_S(a(b, c))=\alpha (F_G(b), F_S(c))$.

• (Recognizing group and set elements.) There are polynomial-time computable functions $C_G$ and $C_S$, such that $C_G(b)=1$ iff $F_G(b)\ne \bot$, and $C_S(b)=1$ iff $F_S(b)\ne \bot$.

• (Random sampling of group and set elements.) There are polynomial-time computable functions $R_G$ and $R_S$, such that $R_G$ uniformly samples a group element $g\in G$, represented by the unique $b\in \{0, 1\}^{p(n)}$ with $F_G(b)=g$, and $R_S$ uniformly samples a set element $s\in S$, represented by some $b\in \{0, 1\}^{q(n)}$ with $F_S(b)=s$.

Remark 1

Some remarks are due for the above model.

• 1. The differences with Brassard and Yung are: (1) allowing infinite groups and sets; (2) adding random sampling of set elements. Note that in the case of infinite groups and sets, the parameters p(n) and q(n) are used to control the bit lengths for the descriptions of legitimate group and set elements. This allows us to incorporate e.g. the lattice isomorphism problem [48] into this framework. In the rest of this article, however, we will mostly work with finite groups and sets, unless otherwise stated.

• 2. The main reason to consider infinite groups is the uses of lattice isomorphism and equivalence of integral bilinear forms in the cryptographic setting.

• 3. The key difference with Couveignes lies in Couveignes’s focus on transitive abelian group actions with trivial stabilizers.

• 4. It is possible to adapt the above framework to use the black-box group model by Babai and Szemerédi [8], whose motivation was to deal with non-unique encodings of group elements (like quotient groups). For our purposes, it is more convenient and practical to assume that the group elements have unique encodings.

• 5. Babai [4] gives an efficient Monte Carlo algorithm for sampling a group element of a finite group in a very general setting which is applicable to most of our instantiations with finite groups.

2.3 The Isomorphism Problem and the One-Way Assumption

Now that we have defined group actions and a computational model, let us examine the isomorphism problems associated with group actions.

Definition 1 (The isomorphism problem)

Let $\alpha :G\times S\rightarrow S$ be a group action. The isomorphism problem for $\alpha$ is to decide, given $s, t\in S$, whether s and t lie in the same orbit under $\alpha$. If they are, the search version of the isomorphism problem further asks to compute some $g\in G$, such that $\alpha (g, s)=t$.

If we assume that there is a distribution on S and we require the algorithm to succeed for (s, t) where s is sampled from this distribution and t is arbitrary, then this is the average-case setting of the isomorphism problem. For example, the first average-case efficient algorithm for the graph isomorphism problem was designed by Babai, Erdős and Selkow in the 1970’s [7].

The hardness of the isomorphism problem provides us with the basic intuition for its use in cryptography. But for cryptographic uses, the promised search version of the isomorphism problem is more relevant, as already observed by Brassard and Yung [17]. That is, suppose we are given $s, t\in S$ with the promise that they are in the same orbit, the problem asks to compute $g\in G$ such that $g\cdot s=t$. Making this more precise and suitable for cryptographic purposes, we formulate the following problem.

Definition 2

(The group-action inversion (GA-Inv) problem). Let $\mathcal {G}$ be a group action family, such that for a security parameter $\lambda$, $\mathcal {G}(1^\lambda )$ consists of descriptions of a group G, a set S with $\log (|G|)={{\,\mathrm{\mathrm {poly}}\,}}(\lambda )$, $\log (|S|)={{\,\mathrm{\mathrm {poly}}\,}}(\lambda )$, and an group action $\alpha : G\times S \rightarrow S$ that can be computed efficiently, which we denote as a whole as a public parameter $\texttt {params}$. Generate random $s\leftarrow S$ and $g\leftarrow G$, and compute $t: = \alpha (g,s)$. The group-action inversion (GA-Inv) problem is to find g given (s, t).

Definition 3 (Group-action inversion game)

The group-action inversion game is the following game between a challenger and an arbitrary adversary $\mathcal {A}$:

• 1. The challenger and adversary $\mathcal {A}$ agree on the public parameter $\texttt {params}$ by choosing it to be $\mathcal {G}(1^\lambda )$ for some security parameter $\lambda$.

• 2. Challenger samples $s\leftarrow S$ and $g\leftarrow G$ using $R_S$ and $R_G$, computes $t = g \cdot s$, and gives (s, t) to $\mathcal {A}$.

• 3. The adversary $\mathcal {A}$ produces some $g'$ and sends it to the challenger.

• 4. We define the output of the game $\textsf {GA\text {-}Inv} _{\mathcal {A},\mathcal {G}}(1^\lambda ) = 1$ if $g' \cdot s = t$, and say $\mathcal {A}$ wins the game if $\textsf {GA\text {-}Inv} _{\mathcal {A},\mathcal {G}}(1^\lambda ) = 1$.

Definition 4

We say that the group-action inversion (GA-Inv) problem is hard relative to $\mathcal {G}$, if for any polynomial time quantum algorithm $\mathcal {A}$,

$$\begin{aligned} \Pr \bigl [ \textsf {GA\text {-}Inv} _{\mathcal {A},\mathcal {G}}(1^\lambda ) \bigr ] \le {{\,\mathrm{\mathrm {negl}}\,}}(\lambda ). \end{aligned}$$

We propose our first cryptographic assumption in the following. It generalizes the one in [17].

Assumption 1

(One-way group action (OWA) assumption). There exists a family $\mathcal {G}$ relative to which the $\textsf {GA\text {-}Inv}$ problem is hard.

We informally call the group action family $\mathcal {G}$ in Assumption 1 a one-way group action. Its name comes from the fact that, as already suggested in [17], this assumption immediately implies that we can treat $\varGamma _s: G \rightarrow S$ given by $\varGamma _s(g)=\alpha (g, s)$ as a one-way function for a random s. In fact, OWA assumption is equivalent to the assertion that the function $\varGamma :G\times S\rightarrow S\times S$ given by $\varGamma (g, s)=(g\cdot s, s)$ is one-way in the standard sense.

Note that the $\mathrm {OWA}$ assumption comes with the promise that s and t are in the same orbit. The question is to compute a group element that sends s to t. Comparing with Definition 1, we see that the $\mathrm {OWA}$ assumption is stronger than the assumption that the search version of the isomorphism problem is hard for a group action, while incomparable with the decision version. Still, most algorithms for the isomorphism problem we are aware of do solve the search version.

Remark 2

Note that Assumption 1 has a slight difference with that of Brassard and Yung as follows. In [17], Brassard and Yung asks for the existence of some $s\in S$ as in Definition 2, such that for a random $g\in G$, it is not feasible to compute $g'$ that sends s to $\alpha (g, s)$. Here, we relax this condition, namely a random $s\in S$ satisfies this already. One motivation for Brassard and Yung to fix s was to take into account of graph isomorphism, for which Brassard and Crepéau defined the notion of “hard graphs” which could serve as this starting point [16]. However, by Babai’s algorithm [5] we know that hard graphs could not exist. Here we use a stronger notion by allowing a random s, which we believe is a reasonable requirement for some concrete group actions discussed in Sect. 3.

A useful fact for the GA-Inv problem is that it is self-reducible to random instances within the orbit of the input pair. For any given s, let $O_s$ be the orbit of s under the group action $\alpha$. If there is an efficient algorithm $\mathcal {A}$ that computes g from $(t, t')$ where $t' = \alpha (g,t)$ for at least $1/{{\,\mathrm{\mathrm {poly}}\,}}(\lambda )$ fraction of the pairs $(t,t') \in O_s \times O_s$, then the GA-Inv problem can be computed for any $(t,t') \in O_s \times O_s$ with probability $1-e^{-{{\,\mathrm{\mathrm {poly}}\,}}(\lambda )}$. On input $(t,t')$, the algorithm samples random group elements $h,h'$ and calls $\mathcal {A}$ with $(\alpha (h,t),\alpha (h',t'))$. If $\mathcal {A}$ successfully returns g, the algorithm outputs $h^{-1}gh'$ and otherwise repeats the procedure for polynomial number of times.

The one-way assumption leads to several basic cryptographic applications as described in the literature. First, it gives a identification scheme by adapting the zero-knowledge proof system for graph isomorphism [42]. Then via the celebrated Fiat-Shamir transformation [37], one also obtains a signature scheme. Proving quantum security of these protocols, however, would need more care. Detailed proofs may be found in the full version of this paper [53].

3 General Linear Actions on Tensors: The One-Way Group Action Assumption

In this section, we propose the general linear actions on tensors, i.e., the tensor isomorphism problem, as our choice of candidate for the $\mathrm {OWA}$ assumption. We first reflect on what would be needed for a group action to be a good candidate.

3.1 Requirements for a Group Action to Be One-Way

Naturally, the hardness of the $\textsf {GA\text {-}Inv}$ problem for a specific group action needs to be examined in the context of the following four types of algorithms.

• Practical algorithms: implemented algorithms with practical performance evaluations but no theoretical guarantees;

• Average-case algorithms: for some natural distribution over the input instances, there is an algorithm that are efficient for most input instances from this distribution with provable guarantees;

• Worst-case algorithms: efficient algorithms with provable guarantees for all input instances;

• Quantum algorithms: average-case or worst-case efficient algorithms in the quantum setting.

Here, efficient means sub-exponential, and most means $1-1/{{\,\mathrm{\mathrm {poly}}\,}}(n)$ fraction. It is important to keep in mind all possible attacks by these four types of algorithms. Past experience suggests that one problem may look difficult from one viewpoint, but turns out to be easy from another.

The graph isomorphism problem has long been thought to be a difficult problem from the worst-case viewpoint. Indeed, a quasipolynomial-time algorithm was only known very recently, thanks to Babai’s breakthrough [5]. However, it has long been known to be effectively solvable from the practical viewpoint [63, 64]. This shows the importance of practical algorithms when justifying a cryptographic assumption.

Patarin proposed to use polynomial map isomorphism problems in his instantiation of the identification and signature schemes [70]. He also proposed the one-sided version of such problems, which has been studied intensively, mostly from the viewpoint of practical cryptanalysis [11, 14, 15, 35, 41, 55, 61, 71, 72, 74]. However, the problem of testing isomorphism of quadratic polynomials with one secret was recently shown to be solvable in randomized polynomial time [50], using ideas including efficient algorithms for computing the algebra structure, and the $*$-algebra structure underlying such problems. Hence, the investigation of theoretical algorithms is also valuable.

Considering of quantum attacks is necessary for security in the quantum era. Shor’s algorithm, for example, invalidates the hardness assumption of the discrete logarithm problems.

Guided by the difficulty met by the hidden subgroup approach on tackling graph isomorphism [45], Moore, Russell, and Vazirani proposed the code equivalence problem as a candidate for the one-way assumption [65]. However, this problem turns out to admit an effective practical algorithm by Sendrier [77].

One-Way Group Action Assumption and the Hidden Subgroup Approach. From the post-quantum perspective, a general remark can be made on the $\mathrm {OWA}$ assumption and the hidden subgroup approach in quantum algorithm design.

Recall that the hidden subgroup approach is a natural generalization of Shor’s quantum algorithms for discrete logarithm and factoring [79], and can accommodate both lattice problems [75] and isomorphism testing problems [45]. The survey paper of Childs and van Dam [24] contains a nice introduction to this approach.

A well-known approach to formulate GA-Inv as an HSP problem is the following [24, Sec. VII.A]. Let $\alpha :G\times S\rightarrow S$ be a group action. Given $s, t\in S$ with the promise that $t=g\cdot s$ for some $g\in G$, we want to compute g. To cast this problem as an HSP instance, we first formulate it as an automorphism type problem. Let $\tilde{G}=G\wr {{\,\mathrm{\mathrm {S}}\,}}_2$, where ${{\,\mathrm{\mathrm {S}}\,}}_2$ is the symmetric group on two elements, and $\wr$ denotes the wreath product. The action $\alpha$ induces an action $\beta$ of $\tilde{G}$ on $S\times S$ as follows. Given $(g, h, i)\in \tilde{G}=G\wr {{\,\mathrm{\mathrm {S}}\,}}_2$ where $g, h\in G, i\in {{\,\mathrm{\mathrm {S}}\,}}_2$, if i is the identity, it sends $(s, t)\in S\times S$ to $(g\cdot s, h\cdot t)$; otherwise, it sends (s, t) to $(h\cdot t, g\cdot s)$. Given $(s, t)\in S\times S$, we define a function $f_{(s, t)}:\tilde{G}\rightarrow S\times S$, such that $f_{(s, t)}$ sends (g, h, i) to $(g, h, i)\cdot (s, t)$, defined as above. It can be verified that $f_{(s, t)}$ hides the coset of the stabilizer group of (s, t) in $\tilde{G}$. Since s and t lie in the same orbit, any generating set of the stabilizer group of (s, t) contains an element of the form (g, h, i), where i is not the identity element in ${{\,\mathrm{\mathrm {S}}\,}}_2$, $g\cdot s=t$, and $h\cdot t=s$. In particular, g is the element required to solve the GA-Inv problem. In the above reduction to the HSP problem, the ambient group is $G\wr {{\,\mathrm{\mathrm {S}}\,}}_2$ instead of the original G. In some cases like the graph isomorphism problem, because of the polynomial-time reduction from isomorphism testing to automorphism problem, we can retain the ambient group to be G. However, such a reduction is not known for $\mathrm {GLAT}$.

There has been notable progress on the HSP problems for various ambient groups, but the dihedral groups and the symmetric groups have withstood the attacks so far. Indeed, one source of confidence on using lattice problems in post-quantum cryptography lies in the lack of progress in tackling the hidden subgroup problem for dihedral groups [75]. There is formal negative evidence for the applicability of this approach for certain group actions where the groups have high-dimensional representations, like ${{\,\mathrm{\mathrm {S}}\,}}_n$ and ${{\,\mathrm{\mathrm {GL}}\,}}(n, q)$ in the case of the graph isomorphism problem [45] and the permutation code equivalence problem [34]. The general lesson is that current quantum algorithmic technologies seem incapable of handling groups which have irreducible representations of high dimensions.

As mentioned, the $\mathrm {OWA}$ assumption has been discussed in post-quantum cryptography with the instantiation of the permutation code equivalence problem [33, 34, 65, 78]. Though this problem is not satisfying enough due to the existence of effective practical algorithms [77], the following quoted from [65] would be applicable to our choice of candidate to the discussed below.

The design of efficient cryptographic primitives resistant to quantum attack is a pressing practical problem whose solution can have an enormous impact on the practice of cryptography long before a quantum computer is physically realized. A program to create such primitives must necessarily rely on insights into the limits of quantum algorithms, and this paper explores consequences of the strongest such insights we have about the limits of quantum algorithms.

3.2 The Tensor Isomorphism Problem and Others

We now formally define the tensor isomorphism problem and other isomorphism testing problems. For this we need some notation and preparations.

Notation and Preliminaries. We usually use $\mathbb {F}$ to denote a field. The finite field with q elements and the real number field are denoted by $\mathbb {F}_q$ and $\mathbb {R}$, respectively. The linear space of m by n matrices over $\mathbb {F}$ is denoted by $\mathrm {M}(m,n,\mathbb {F})$, and $\mathrm {M}(n, \mathbb {F}):=\mathrm {M}(n, n, \mathbb {F})$. The identity matrix in $\mathrm {M}(n, \mathbb {F})$ is denoted by $I_n$. For $A\in \mathrm {M}(m, n, \mathbb {F})$, $A^t$ denotes the transpose of A. The group of n by n invertible matrices over $\mathbb {F}$ is denoted by ${{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$. We will also meet the notation ${{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {Z})$, the group of n by n integral matrices with determinant $\pm 1$. We use a slightly non-standard notation ${{\,\mathrm{\mathrm {GL}}\,}}(m, n, \mathbb {F})$ to denote the set of rank $\min (m, n)$ matrices in $\mathrm {M}(m, n, \mathbb {F})$. We use $\langle \cdot \rangle$ to denote the linear span; for example, given $A_1, \dots , A_k\in \mathrm {M}(m, n, \mathbb {F})$, $\langle A_1,\dots , A_k\rangle$ is a subspace of $\mathrm {M}(m, n, \mathbb {F})$.

We will meet some subgroups of ${{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$ as follows. The symmetric group ${{\,\mathrm{\mathrm {S}}\,}}_n$ on n objects is embedded into ${{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$ as permutation matrices. The orthogonal group ${{\,\mathrm{\mathrm {O}}\,}}(n, \mathbb {F})$ consists of those invertible matrices A such that $A^tA=I_n$. The special linear group ${{\,\mathrm{\mathrm {SL}}\,}}(n, \mathbb {F})$ consists of those invertible matrices A such that $\det (A)=1$. Finally, when $n=\ell ^2$, there are subgroups of ${{\,\mathrm{\mathrm {GL}}\,}}(\ell ^2, \mathbb {F})$ isomorphic to ${{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})$. This can be seen as follows. First we fix an isomorphism of linear spaces $\phi : \mathbb {F}^{\ell ^2}\rightarrow \mathrm {M}(\ell , \mathbb {F})$¹. Then $\mathrm {M}(\ell , \mathbb {F})$ admits an action by ${{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})$ by left and right multiplications, e.g. $(A, D)\in {{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})$ sends $C\in \mathrm {M}(\ell , \mathbb {F})$ to $ACD^t$. Now use $\phi ^{-1}$ and we get one subgroup of ${{\,\mathrm{\mathrm {GL}}\,}}(\ell ^2, \mathbb {F})$ isomorphic to ${{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})$.

Definitions of Several Group Actions. We first recall the concept of tensors and the group actions on the space of k-tensors as introduced in Sect. 1.

Definition 5 (Tensor)

A k-tensor T of local dimensions $d_1, d_2, \ldots , d_k$ over $\mathbb {F}$, written as

$$\begin{aligned} T = (T_{i_1, i_2, \ldots , i_k}), \end{aligned}$$

is a multidimensional array with k indices and its components $T_{i_1, i_2, \ldots , i_k}$ chosen from $\mathbb {F}$ for all $i_j \in \{1, 2, \ldots , d_j\}$. The set of k-tensors of local dimensions $d_1, d_2, \ldots , d_k$ over $\mathbb {F}$ is denoted as

$$\begin{aligned} \mathrm {T}(d_1, d_2, \ldots , d_k, \mathbb {F}). \end{aligned}$$

The integer k is called the order of tensor T.

Group Action 1

(The general linear group action on tensors). Let $\mathbb {F}$ be a field, k, $d_1, d_2, \ldots , d_k$ be integers.

• Group G: $\prod _{j=1}^k {{\,\mathrm{\mathrm {GL}}\,}}(d_j, \mathbb {F})$.

• Set S: $\mathrm {T}(d_1, d_2, \ldots , d_k, \mathbb {F})$.

• Action $\alpha$: for a k-tensor $T \in S$, a member $M = (M^{(1)}, M^{(2)}, \ldots , M^{(k)})$ of the group G,

  $$\begin{aligned} \alpha (M, T) = \biggl ( \bigotimes _{j=1}^k M^{(j)} \biggr ) T = \sum _{l_1, l_2, \ldots , l_k} \biggl ( \prod _{j=1}^k M^{(j)}_{i_j,l_j} \biggr ) T_{l_1, l_2, \ldots , l_k}. \end{aligned}$$

We refer to the general linear group action on tensors in Action 1 as $\mathrm {GLAT}$. In the following, let us formally define several problems which have been referred to frequently in the above discussions.

As already observed by Brassard and Yung [17], the discrete logarithm problem can be formulated using the language of group actions. More specifically, we have:

Group Action 2

(Discrete Logarithm in Cyclic Groups of Prime Orders). Let p be a prime, $\mathbb {Z}_p$ the integer.

• Group G: $\mathbb {Z}_p^{*}$, the multiplicative group of units in $\mathbb {Z}_p$.

• Set S: $C_p\setminus \{\mathrm {id}\}$, where $C_p$ is a cyclic group of order p and $\mathrm {id}$ is the identity element.

• Action $\alpha$: for $a\in \mathbb {Z}_p^{*}$, and $s\in S$, $\alpha (a, s)=s^a$.

Note that in the above, we refrained from giving a specific realization of the cyclic group $C_p$ for the sake of clarify; the reader may refer to Boneh’s excellent survey [13] for concrete proposals that can support the security of the Decisional Diffie-Hellman assumption.

The linear code permutation equivalence (LCPE) problem asks to decide whether two linear codes (i.e. linear subspaces) are the same up to a permutation of the coordinates. It has been studied in the coding theory community since the 1990’s [73, 77].

Group Action 3

(Group action for Linear Code Permutation Equivalence problem (LCPE)). Let m, d be integers, $m\le d$, and let $\mathbb {F}$ be a field.

• Group G: ${{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F}) \times {{\,\mathrm{\mathrm {S}}\,}}_d$.

• Set S: ${{\,\mathrm{\mathrm {GL}}\,}}(m, d, \mathbb {F})$.

• Action $\alpha$: for $A \in S$, $M = (N,P) \in G$, $\alpha (M, A) = N A P^t$.

The connection with coding theory is that A can be viewed as the generating matrix of a linear code (a subspace of $\mathbb {F}_q^n$), and N is the change of basis matrix taking care of different choices of bases. Then, P, as a permutation matrix, does not change the weight of a codeword— that is a vector in $\mathbb {F}^n$. (There are other operations that preserve weights [78], but we restrict to consider this setting for simplicity.) The GA-Inv problem for this group action is called the linear code permutation equivalence (LCPE) problem, which has been studied in the coding theory community since the 1980’s [57], and we can dodge the only successful attack [77] by restricting to self-dual codes.

The following group action induces a problem called the polynomial isomorphism problems proposed by Patarin [70], and has been studied in the multivariate cryptography community since then.

Group Action 4

(Group action for the Isomorphism of Quadratic Polynomials with two Secrets problem (IQP2S)). Let m, d be integers and $\mathbb {F}$ a finite field.

• Group G: ${{\,\mathrm{\mathrm {GL}}\,}}(d, \mathbb {F}) \times {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})$.

• Set S: The set of tuples of homogeneous polynomials $(f_1, f_2, \ldots , f_m)$ for $f_i \in \mathbb {F}[x_1, x_2, \ldots , x_d]$ the polynomial ring of d variables over $\mathbb {F}$.

• Action $\alpha$: for $f = (f_1, f_2, \ldots , f_m) \in S$, $M = (C, D) \in G$, $C'=C^{-1}$, define $\alpha (M, f) = (g_1, g_2, \ldots , g_m)$ by $g_i(x_1, x_2, \ldots , x_d)=\sum _{j=1}^m D_{i,j} f_i(x_1', \dots , x_d')$, where $x_i'=\sum _{j=1}^d C'_{i,j}x_j$.

The GA-Inv problem for this group action is essentially the isomorphism of quadratic polynomials with two secrets (IQP2S) assumption. The algebraic interpretation here is that the tuple of polynomials $(f_1, \dots , f_n)$ is viewed as a polynomial map from $\mathbb {F}^n$ to $\mathbb {F}^m$, by sending $(a_1, \dots , a_n)$ to $(f_1(a_1, \dots , a_n), \dots , f_m(a_1, \dots , a_n))$. The changes of bases by C and D then are naturally interpreted as saying that the two polynomial maps are essentially the same.

Finally, the GA-Inv problem for the following group action originates from computational group theory, and is basically equivalent to a bottleneck case of the group isomorphism problem (i.e. p-groups of class 2 and exponent p) [59, 69].

Group Action 5

(Group action for alternating matrix space isometry (AMSI)). Let d, m be integers and $\mathbb {F}$ be a finite field.

• Group G: ${{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})$.

• Set S: the set of all linear spans $\mathcal {A}$ of d alternating² matrices $A_i$ of size $m\times m$.

• Action $\alpha$: for $\mathcal {A}= \langle A_1, A_2, \ldots , A_d \rangle \in S$, $C \in G$, $\alpha (C, \mathcal {A}) = \langle B_1, B_2, \ldots , B_d \rangle$ where $B_i = C A_i C^t$ for all $i=1, 2, \ldots , d$.

3.3 General Linear Actions on Tensors as One-Way Action Candidates

The Central Position of 3-tensor Isomorphism. As mentioned, the four problems, linear code permutation equivalence (LCPE), isomorphism of polynomials with two secrets (IQP2S), and alternating matrix space isometry (AMSI), have been studied in coding theory, multivariate cryptography, and computational group theory, respectively, for decades. Only recently we begin to see connections among these problems which go through the 3TI problem thanks to the work of Futorny, Grochow, and Sergeichuk [39]. We spell out this explicitly.

Observation 2

([39, 43]). IQP2S, AMSI, GI, and LCPE reduce to 3TI.

Proof

Note that the set underlying Group Action 5 consists of d-tuples of $m\times m$ alternating matrices. We can write such a tuple $(A_1, \dots , A_d)$ as a 3-tensor A of dimension $m\times m\times d$, such that $A_{i,j,k}=(A_k)_{i,j}$. Then AMSI asks to test whether two such 3-tensors are in the same orbit under the action of $(M, N)\in {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(d, \mathbb {F})$ by sending a 3-tensor A to the result of applying (M, M, N) to A as in the definition of $\mathrm {GLAT}$.

Such an action belongs to the class of actions on 3-tensors considered in [39] under the name linked actions. This work constructs a function r from 3-tensors to 3-tensors, such that A and B are in the same orbit under ${{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(d, \mathbb {F})$ if and only if r(A) and r(B) are in the same orbit under ${{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(d, \mathbb {F})$. This function r can be computed efficiently [39, Remark 1.1].

This explains the reduction of the isomorphism problem for Group Action 5 to the 3-tensor isomorphism problem. For Group Action 4, by using the classical correspondence between homogeneous quadratic polynomials and symmetric matrices, we can cast it in a form similar to Group Action 5, and then apply the above reasoning using again [39].

Finally, to reduce the graph isomorphism problem (GI) and the linear code permutation equivalent problem (LCPE) to the 3-tensor isomorphism problem, we only need to take care of LCPE as GI reduces to LCPE [73]. To reduce LCPE to 3TI, we can reduce it to the matrix Lie algebra conjugacy problem by [43], which reduces to 3TI by [39] along the linked action argument, though this time linked in a different way.    $\square$

This put 3TI at a central position of these difficult isomorphism testing problems arising from multivariate cryptography, computational group theory, and coding theory. In particular, from the worst-case analysis viewpoint, 3TI is the hardest problem among all these. This also allows us to draw experiences from previous research in various research communities to understand 3TI.

Current Status of the Tensor Isomorphism Problem and Its One-Way Action Assumption. We now explain the current status of the tensor isomorphism problem to support it as a strong candidate for the $\mathrm {OWA}$ assumption. Because of the connections with isomorphism of polynomials with two secrets (IQP2S) and alternating matrix space isometry (AMSI), we shall also draw results and experiences from the multivariate cryptography and the computational group theory communities.

For convenience, we shall restrict to finite fields $\mathbb {F}_q$, though other fields are also interesting. That is, we consider the action of ${{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F}_q)\times {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F}_q) \times {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F}_q)$ on $T \in \mathrm {T}(\ell , n, m, \mathbb {F}_q)$. Without loss of generality, we assume $\ell \ge n \ge m$. The reader may well think of the case when $\ell = n = m$, which seems to be the most difficult case in general. Correspondingly, we will assume that the instances for IQP2S are m-tuples of homogeneous quadratic polynomials in n variables over $\mathbb {F}_q$, and the instances for AMSI are m-tuples of alternating matrices of size $n\times n$ over $\mathbb {F}_q$.

To start, we note that 3TI over finite fields belongs to $\mathrm {NP} \cap \mathrm {coAM}$, following the same $\mathrm {coAM}$-protocol for graph isomorphism.

For the worst-case time complexity, it can be solved in time $q^{m^2}\cdot {{\,\mathrm{\mathrm {poly}}\,}}(\ell , m, n, \log q)$, by enumerating ${{\,\mathrm{\mathrm {GL}}\,}}(m, q)$, and then solving an instance of the matrix tuple equivalence problem, which asks to decide whether two matrix tuples are the same under the left-right multiplications of invertible matrices. This problem can be solved in deterministic polynomial time by reducing [50] to the module isomorphism problem, which in turn admits a deterministic polynomial-time solution [18, 25, 51]. It is possible to reduce the complexity to $q^{c m^2}\cdot {{\,\mathrm{\mathrm {poly}}\,}}(\ell , m, n, \log q)$ for some constant $0<c<1$, by using some dynamic programming technique as in [59]. But in general, the worst-case complexity could not go beyond this at present, which matches the experiences of IQP2S and AMSI as well; see [50].

For the average-case time complexity, it can be solved in time $q^{O(m)}\cdot {{\,\mathrm{\mathrm {poly}}\,}}(\ell , n)$, by adapting the average-case algorithm for AMSI in [59]. This also matches the algorithm for IQP2S which has an average-case running time of $q^{O(n)}$ [15].

For practical algorithms, we draw experiences from the computational group theory community and the multivariate cryptography community. In the computational group theory community, the current status of the art is that one can hope to handle 10-tuples of alternating matrices of size $10\times 10$ over $\mathbb {F}_{13}$, but absolutely not, for 3-tensors of local dimension say 100, even though in this case the input can still be stored in only a few megabytes.³ In the multivariate cryptography community, the Gröbner basis technique [35] and certain combinatorial technique [15] have been studied to tackle IQF2S problem. However, these techniques are not effective enough to break it [15]⁴.

For quantum algorithms, 3TI seems difficult for the hidden subgroup approach, due to the reasons presented in Sect. 3.1.

Finally, let us also elaborate on the prospects of using those techniques for graph isomorphism [5] and for isomorphism of quadratic polynomials with one secret [50] to tackle 3TI. In general, the difficulties of applying these techniques seem inherent.

We first check out the graph isomorphism side. Recall that most algorithms for graph isomorphism, including Babai’s [5], are built on two families of techniques: group-theoretic, and combinatorial. To use the group-theoretic techniques, we need to work with matrix groups over finite fields instead of permutation groups. Algorithms for matrix groups over finite fields are in general far harder than those for permutation groups. For example, the basic membership problem is well-known to be solvable by Sims’s algorithm [80], while for matrix groups over finite fields of odd order, this was only recently shown to be efficiently solvable with a number-theoretic oracle and the algorithm is much more involved [6]. To use the combinatorial techniques, we need to work with linear or multilinear structures instead of combinatorial structures. This shift poses severe limitations on the use of most combinatorial techniques, like individualizing a vertex. For example, it is quite expensive to enumerate all vectors in a vector space over a finite field, while this is legitimate to go over all elements in a set.

We then check out the isomorphism of quadratic polynomials with one secret side. The techniques for settling this problem as in [50] are based on those developed for the module isomorphism problem [18, 25, 51], involutive algebras [84], and computing algebra structures [38]. The starting point of that algorithm solves an easier problem, namely testing whether two matrix tuples are equivalent under the left-right multiplications. That problem is essentially linear, so the techniques for the module isomorphism problem can be used. After that we need to utilize the involutive algebra structure [84] based on [38]. However, for 3TI, there is no such easier linear problem to start with, so it is not clear how those techniques can be applied.

To summarize, the 3-tensor isomorphism problem is difficult from all the four types of algorithms mentioned in Sect. 3.1. Furthermore, the techniques in the recent breakthrough on graph isomorphism [5], and the solution of the isomorphism of quadratic polynomials with one secret [50], seem not applicable to this problem. All these together support this problem as a strong candidate for the one-way assumption.

Choices of the Parameters. Having reviewed the current status of the tensor isomorphism problem, we lay out some principles of choosing the parameters for the security, namely the order k, the dimensions $d_i$, and the underlying field $\mathbb {F}$.

Let us first explain why we focus on $k=3$, namely 3-tensors. Of course, k needs to be $\ge$3 as most problems about 2-tensors, i.e. matrices, are easy. Recently, Grochow and the third author show that k-tensor isomorphism reduces to 3-tensor isomorphism [44]. This justifies our choice of $k=3$ from the worst-case analysis viewpoint. From the practical viewpoint though, it will be interesting to investigate into the tradeoff between the local dimensions $d_i$ and k.

After fixing $k=3$, it is suggested to set $d_1=d_2=d_3$. This is because of the argument when examining the worst-case time complexity in the above subsection.

Then for the underlying finite field $\mathbb {F}_q$, the intuition is that setting q to be a large prime would be more secure. Note that we can still store an exponentially large prime using polynomially-many bits. This is because, if q is small, then the “generic” behaviors as ensured by the Lang–Weil type theorems [56] may not be that generic. So some non-trivial properties may arise which then help with isomorphism testing. This is especially important for the pseudorandom assumption to be discussed Sect. 4. We then examine whether we want to set q to be a large prime, or a large field with a small characteristic. The former one is preferred, because the current techniques in computer algebra and computational group theory, cf. [50] and [6], can usually work efficiently with large fields of small characteristics.

However, let us emphasize that even setting q to be a constant, we do not have any concrete evidence for breaking $\mathrm {GLAT}$ as a one-way group action candidate. Furthermore, there are certain problems that are easy over large fields, while NP-hard over small fields; one such example is the maximum rank problem for matrix spaces [21]. To summarize, the above discussion on the field size issue is rather hypothetical and conservative.

4 The Pseudorandom Action Assumption

In this section, we introduce the new security assumption for group actions, namely pseudorandom group actions, which generalizes the Decisional Diffie-Hellman assumption. In Sect. 5, we shall study the prospect of using the general linear action on tensors as a candidate for this assumption. In the full version of this paper [53], the reader can find the cryptographic uses of this assumption including signatures and pseudorandom functions.

Definition 6

Let $\mathcal {G}$ be a group family as specified before. Choose public parameters $\texttt {params}= (G,S,\alpha )$ to be $\mathcal {G}(1^\lambda )$. Sample $s\leftarrow S$ and $g\leftarrow G$. The group action pseudorandomness (GA-PR) problem is that given (s, t), where $t = \alpha (g, s)$ or $t\leftarrow S$, decide which case t is sampled from.

Definition 7 (Pseudorandom group action game)

The pseudorandom group action game is the following game between a challenger and an adversary $\mathcal {A}$:

• The challenger and the adversary $\mathcal {A}$ agree on the public parameters $\texttt {params}= (G, S, \alpha )$ by choosing it to be $\mathcal {G}(1^\lambda )$ for some security parameter $\lambda$.

• Challenger samples random bit $b\in \{0,1\}$, $s\leftarrow S$, $g\leftarrow G$, and chooses $t\leftarrow S$ if $b=0$ and $t = g \cdot s$ if $b=1$.

• Give (s, t) to $\mathcal {A}$ who produces a bit $a\in \{0,1\}$.

• We define the output of the game $\textsf {GA\text {-}PR} _{\mathcal {A},\mathcal {G}}(1^\lambda ) = 1$ and say $\mathcal {A}$ wins the game if $a=b$.

Definition 8

We say that the group-action pseudorandomness (GA-PR) problem is hard relative to $\mathcal {G}$, if for any polynomial-time quantum algorithm $\mathcal {A}$,

$$\begin{aligned} \Pr [ \textsf {GA\text {-}PR} _{\mathcal {A},\mathcal {G}}(1^\lambda ) = 1 ] = {{\,\mathrm{\mathrm {negl}}\,}}(\lambda ). \end{aligned}$$

Some remarks on this definition are due here.

For Transitive and Almost Transitive Actions. In the case of transitive group actions, as an easy corollary of Observation 1, we have the following.

Observation 3

GA-PR  problem is hard, if the group action $\alpha$ is transitive.

Indeed, when $\alpha$ is transitive, the two distributions in Definition 6 are the same, so in fact statistically impossible to distinguish.

Slightly generalizing the transitive case, it is not hard to see that GA-PR  problem is hard, if there exists a “dominant” orbit $O\subseteq S$. Intuitively, this means that O is too large such that random s and t from S would both lie in O with high probability. For example, consider the action of ${{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$ on $\mathrm {M}(n, \mathbb {F})$ by the left and right multiplications. The orbits are determined by the ranks of matrices in $\mathrm {M}(n, \mathbb {F})$, and the orbit of matrices of full-rank is dominant. But again, such group actions do not seem very useful for cryptographic purposes. Indeed, we require the orbit structure to satisfy that random s and t do not fall into the same orbit. Let us formally put forward this condition.

Definition 9

We say that a group action $\alpha$ of G on S does not have a dominant orbit, if

$$\begin{aligned} \Pr _{s,t\leftarrow S}\, [s, t \text { lie in the same orbit}] = {{\,\mathrm{\mathrm {negl}}\,}}(\lambda ). \end{aligned}$$

This definition is closely related to a classical question in geometry, namely classifying representations with a Zariski-dense orbit. When the group is a connected linear algebraic group over $\mathbb {C}$ and the representation is irreducible, this question has been settled by Sato and Kimura [76].

We now put forward a key assumption.

Assumption 2

(Pseudorandom group action (PRA) assumption). There exists an $\mathcal {G}$ outputting a group action without a dominant orbit, relative to which the $\textsf {GA\text {-}PR}$ problem is hard.

The name comes from the fact that the PRA assumption says ‘in spirit’ that the function $\varGamma :G\times S\rightarrow S\times S$ given by $\varGamma (g, s)=(g\cdot s, s)$ is a secure PRG. Here, it is only ‘in spirit’, because the PRA assumption does not include the usual expansion property of the PRG. Rather, it only includes the inexistence of a dominant orbit.

The applications of the $\mathrm {PRA}$ assumption including more efficient quantum-secure digital signature schemes and pseudorandom function constructions are given in the full version of this paper [53].

Subsuming the Classical Diffie-Hellman Assumption. We now formulate the classical decisional Diffie-Hellman (DDH) assumption as an instance of the pseudorandom group action assumption. To see this, we need the following definition.

Definition 10

Let $\alpha :G\times S\rightarrow S$ be a group action. The d-diagonal action of $\alpha$, denoted by $\alpha ^{(d)}$, is the group action of G on $S^d$, the Cartesian product of d copies of S, where $g\in G$ sends $(s_1, \dots , s_d)\in S^d$ to $(g\cdot s_1, \dots , g\cdot s_d)$.

The following observation shows that the classical DDH can be obtained by instantiating GA-PR  with a concrete group action.

Observation 4

Let $\alpha$ be the group action in Group Action 2. The classical Decisional Diffie-Hellman assumption is equivalent to the $\mathrm {PRA}$ assumption instantiated with $\alpha ^{(2)}$, the 2-diagonal action of $\alpha$.

Proof

Recall from Group Action 2 defines an action $\alpha$ of $G\cong \mathbb {Z}_p^*$ on $S=C_p\setminus \{\mathrm {id}\}$ where $C_p$ is a cyclic group of order p. The 2-diagonal action $\alpha ^{(2)}$ is defined by $a\in \mathbb {Z}_p^*$ sending $(s, t)\in S\times S$ to $(s^a, t^a)$. Note that while $\alpha$ is transitive, $\alpha ^{(2)}$ is not, and in fact it does not have a dominant orbit.

$\mathrm {PRA}$ instantiated with $\alpha ^{(2)}$ then asks to distinguish between the following two distributions. The first distribution is $((s, t), (s', t'))$ where $s, t, s', t'\in _R S$. Since $\alpha$ is transitive, by Observation 1, this distribution is equivalent to $((s, s^a), (s^b, s^c))$, where $s\in _R S$ and $a, b, c\in _R G$. The second distribution is $((s, t), (s^b, t^b))$, where $s, t\in _R S$, and $b\in _R G$. Again, by Observation 1, this distribution is equivalent to $((s, s^a), (s^b, s^{ab}))$, where $s\in _R S$, and $a, b\in _R G$.

We then see that this is just the Decisional Diffie-Hellman assumption⁵.    $\square$

As will be explained in Sect. 5.1, the pseudorandom assumption is a strong one, in a sense much stronger than the one-way assumption. Therefore, Observation 4 is important because, by casting the classical Diffie-Hellman assumption as an instance of the pseudorandom assumption, it provides a non-trivial and well-studied group action candidate for this assumption.

Of course, the DDH assumption is no longer secure under quantum attacks. Recently, this assumption in the context of supersingular isogeny based cryptography has been proposed by De Feo and Galbraith in [36]. We will study the possibility for the 3-tensor isomorphism problem as a pseudorandom group action candidate in Sect. 5

The d-Diagonal Pseudorandomness Assumption. Motivated by Observation 4, it will be convenient to specialize $\textsf {GA\text {-}PR}$ to diagonal actions, and make the following assumption.

Definition 11

The d-diagonal pseudorandomness ($\textsf {GA\text {-}PR} (d)$) problem for a group action $\alpha$, is defined to be the pseudorandomness problem for the d-diagonal group action $\alpha ^{(d)}$.

We emphasize that $\textsf {GA\text {-}PR} (d)$ is just $\textsf {GA\text {-}PR}$ applied to group actions of a particular form, so a special case of $\textsf {GA\text {-}PR}$. Correspondingly, we define $\mathrm {PRA} (d)$ to be the assumption that $\textsf {GA\text {-}PR} (d)$ is hard relative to some $\mathcal {G}$.

Given a group action $\alpha :G\times S\rightarrow S$, let $F_\alpha =\{f_g:S\rightarrow S \mid g\in G, f_g(s)=g\cdot s\}$. It is not hard to see that $\mathrm {PRA} (d)$ is equivalent to say that $F_\alpha$ is a d-query weak PRF in the sense of Maurer and Tessaro [62]. This gives a straightforward cryptographic use of the $\mathrm {PRA} (d)$ assumption.

Given $d, e\in \mathbb {Z}^+$, $d<e$, it is clear that $\mathrm {PRA} (e)$ is no weaker than $\mathrm {PRA} (d)$. Indeed, given an algorithm A that distinguishes between

$$((s_1, \dots , s_d), (g\cdot s_1, \dots g\cdot s_d)) \text { and } ((s_1, \dots , s_d), (t_1, \dots , t_d)),$$

where $s_i, t_j\leftarrow S$, and $g\leftarrow G$, one can use A to distinguish between $((s_1, \dots , s_e), (g\cdot s_1, \dots g\cdot s_e))$ and $((s_1, \dots , s_e), (t_1, \dots , t_e))$, by just looking at the first d components in each tuple. It is an interesting question whether $\mathrm {PRA} (e)$ is strictly stronger than $\mathrm {PRA} (d)$. Note though that in the following, we will exhibit some group actions, for which $\mathrm {PRA} (d)$ does not hold for large enough d.

5 General Linear Actions on Tensors: The Pseudorandom Action Assumption

5.1 Requirements for a Group Action to Be Pseudorandom

Clearly, a first requirement for a group action to be pseudorandom is that it should be one-way. Further requirements naturally come from certain attacks. We have devised the following attack strategies. These attacks suggest that the pseudorandom assumption is closely related to the orbit closure intersection problem which has received considerable attention recently.

Isomorphism Testing in the Average-Case Setting. To start with, we consider the impact of an average-case isomorphism testing algorithm on the pseudorandom assumption. Recall that for a group action $\alpha :G\times S\rightarrow S$, an average-case algorithm is required to work for instances (s, t) where $s\leftarrow S$ and t is arbitrary. Let n be the input size to this algorithm. The traditional requirement for an average-case algorithm is that it needs to work for all but at most $1/{{\,\mathrm{\mathrm {poly}}\,}}(n)$ fraction of $s\in S$, like such algorithms for graph isomorphism [7] and for alternating matrix space isometry [59]. However, in order for such an algorithm to break the pseudorandom assumption, it is enough that it works for a non-negligible, say $1/{{\,\mathrm{\mathrm {poly}}\,}}(n)$, fraction of the instances. This is quite relaxed compared to the traditional requirement.

The Supergroup Attack. For a group action $\alpha :G\times S\rightarrow S$, a supergroup action of $\alpha$ is another group action $\beta :H\times S\rightarrow S$, such that (1) G is a subgroup of H, (2) the restriction of $\beta$ to G, $\beta |_G$, is equal to $\alpha$. If it further holds that (3.1) the isomorphism problem for H is easy, and (3.2) $\beta$ is not dominant, we will then have the following so-called supergroup attack. Give input $s,t \in S$, the adversary for the $\textsf {GA\text {-}PR}$ problem of $\alpha$ will use the solver for the isomorphism problem for H to check if s, t are from the same orbit induced by H and return 1 if they are from the same orbit and 0 otherwise. If s, t are from the same orbit induced by G, the adversary always returns the correct answer as G is a subgroup of H. In the case that s, t are independently chosen from S, by the fact that $\beta$ is not dominant, the adversary will return the correct answer 0 with high probability.

The Isomorphism Invariant Attack. Generalizing the condition (3) above, we can have the following more general strategy as follows. We now think of G and H as defining equivalence relations by their orbit structures. Let $\sim _G$ (resp. $\sim _H$) be the equivalence relation defined by G (resp. H). By the conditions (1) and (2), we have (a) $\sim _H$ is coarser than $\sim _G$. By the condition (3.1), we have (b) $\sim _H$ is easy to decide. By the condition (3.2), we have (c) $\sim _H$ have enough equivalence classes. Clearly, if a relation $\sim$, not necessarily defined by a supergroup H, satisfies (a), (b), and (c), then $\sim$ can also be used to break the $\mathrm {PRA}$ assumption for G.

Such an equivalence relation is more commonly known as an isomorphism invariant, namely those properties that are preserved under isomorphism. The sources of isomorphism invariants can be very versatile. The supergroup attack can be thought of as a special case of category where the equivalence relation is defined by being isomorphic under a supergroup action. Another somewhat surprising and rich “invariant” comes from geometry, as we describe now.

The Geometric Attack. In the case of matrix group actions, the underlying vector spaces usually come with certain geometry which can be exploited for the attack purpose. Let $\alpha$ be a group action of G on $V\cong \mathbb {F}^d$. For an orbit $O\subseteq V$, let its Zariski closure be $\overline{O}$. Let $\sim$ be the equivalence relation on V, such that for $s, t\in O$, $s\sim t$ if and only if $\overline{O_s} \cap \overline{O_t} \ne \emptyset$. It is obvious that $\sim$ is a coarser relation than $\sim _G$. Furthermore, except some degenerate settings when m or n are very small, there would be enough equivalence classes defined by $\sim$, because of the dimension reason. So (a) and (c) are satisfied. Therefore, if we could test efficiently whether the orbit closures of s and t intersect, (b) would be satisfied and we could break the $\mathrm {PRA}$ for $\alpha$. This problem, known as the orbit closure intersection problem, has received considerable attention recently.

Another straightforward approach based on this viewpoint is to recall that the geometry of orbit closures is determined by the ring of invariant polynomials [67]. More specifically, the action of G on V induces an action on $\mathbb {F}[V]$, the ring of polynomial functions on V. As $V\cong \mathbb {F}^d$, $\mathbb {F}[V]\cong \mathbb {F}[x_1, \dots , x_d]$. Those polynomials invariant under this induced action form a subring of $\mathbb {F}[V]$, denoted as $\mathbb {F}[V]^G$. If there exists one easy-to-compute, non-trivial, invariant polynomial f from $\mathbb {F}[V]^G$, we could then use f to evaluate on the input instances and distinguish between the random setting (where f is likely to evaluate differently) and the pseudorandom setting (where f always evaluates the same).

Example Attacks. We now list some examples to illustrate the above attacks.

An Example of Using the Isomorphism Invariant Attack. We first consider the isomorphism invariant attack in the graph isomorphism case. Clearly, the degree sequence, consisting of vertex degrees sorted from large to small, is an easy to compute isomorphism invariant. A brief thought suggests that this invariant is already enough to break the pseudorandom assumption for graph isomorphism.

An Example of Using the Geometric Attack. We consider a group action similar to the 3-tensor isomorphism case (Group Action 1), inspired by the quantum marginal problem [19]. Given a 3-tensor of size $\ell \times n\times m$, we can “slice” this 3-tensor according to the third index to obtain a tuple of m matrices of size $\ell$ by n. Consider the action of $G={{\,\mathrm{\mathrm {O}}\,}}(\ell , \mathbb {F})\times {{\,\mathrm{\mathrm {O}}\,}}(n, \mathbb {F})\times {{\,\mathrm{\mathrm {SL}}\,}}(m, \mathbb {F})$ on matrix tuples $\mathrm {M}(\ell \times n, \mathbb {F})^m$, where the three direct product factors act by left multiplication, right multiplication, and linear combination of the m components, respectively. For a matrix tuple $(A_1, \dots , A_m)$ where $A_i\in \mathrm {M}(\ell \times n, \mathbb {F})$, form an $\ell n\times m$ matrix A where the i-th column of A is obtained by straightening $A_i$ according to columns. Then $A^tA$ is an m by m matrix. The polynomial $f=\det (A^tA)$ is then a polynomial invariant for this action. For this note that the group ${{\,\mathrm{\mathrm {O}}\,}}(\ell , \mathbb {F})\times {{\,\mathrm{\mathrm {O}}\,}}(n, \mathbb {F})$ can be embedded as a subgroup of ${{\,\mathrm{\mathrm {O}}\,}}(\ell n, \mathbb {F})$, so its action becomes trivial on $A^tA$. Then the determinant is invariant under the ${{\,\mathrm{\mathrm {SL}}\,}}(m, \mathbb {F})$. When $m<\ell n$, which is the interesting case, $\det (A^tA)$ is non-zero. It follows that we have a non-trivial, easy-to-compute, polynomial invariant which can break the $\mathrm {PRA}$ assumption for this group action.

An Example of Using the Supergroup Attack. We then explain how the supergroup attack invalidates the $\mathrm {PRA} (d)$ assumption for certain families of group actions with $d>1$.

Let $\alpha$ be a linear action of a group G on a vector space $V\cong \mathbb {F}^{N}$. We show that as long as $d>N$, $\mathrm {PRA} (d)$ does not hold. To see this, the action of G on V gives a homomorphism $\phi$ from G to ${{\,\mathrm{\mathrm {GL}}\,}}(V)\cong {{\,\mathrm{\mathrm {GL}}\,}}(N, \mathbb {F})$. For any $g\in G$, and $v_1, \dots , v_d\in V$, we can arrange an $N\times d$ matrix $S=[v_1, \dots , v_d]$, such that $T=[\phi (g)v_1, \dots , \phi (g)v_d]=\phi (g)[v_1, \dots , v_d]$. On the other hand, for $u_1, \dots , u_d\in V$, let $T'=[u_1, \dots , u_d]$. Let us consider the row spans of S, T and $T'$, which are subspaces of $\mathbb {F}^d$ of dimension $\le N<d$. Clearly, the row spans of S and T are the same. On the other hand, when $u_i$’s are random vectors, the row span of $T'$ is unlikely to be the same as that of S. This gives an efficient approach to distinguish between T and $T'$.

We can upgrade the above attack even further as follows. Let $\alpha$ be a linear action of G on the linear space of matrices $M=\mathrm {M}(m\times n, \mathbb {F})$. Recall that ${{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$ acts on M by left and right multiplications. Suppose $\alpha$ gives rise to a homomorphism $\phi :G\rightarrow {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$. For $g\in G$, if $\phi (g)=(A, B)\in {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$, we let $\phi _1(g):=A\in {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})$, and $\phi _2(g)=B\in {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$. We now show that when $d>(m^2+n^2)/(mn)$, $\mathrm {PRA} (d)$ does not hold for $\alpha$. To see this, for any $g\in G$, and $S=(A_1, \dots , A_d)\in \mathrm {M}(m\times n, \mathbb {F})^d$, let

$$T=(\phi _1(g)^tA_1\phi _2(g), \dots , \phi _1(g)^tA_d\phi _2(g)).$$

On the other hand, let $T'=(B_1, \dots , B_d)\in M^d$. Since $\dim (S)=\dim ({{\,\mathrm{\mathrm {GL}}\,}}(m\times n, \mathbb {F})^d)=mnd>m^2+n^2=\dim ({{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})),$ $\alpha$ does not have a dominant orbit (cf. Definition 9) This means that, when $B_i$’s are sampled randomly from S, $T'$ is unlikely to be in the same orbit as S. Now we use the fact that, the isomorphism problem for the action of ${{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})$ on S can be solved in deterministic polynomial time [50, Proposition 3.2]. This gives an efficient approach to distinguish between T and $T'$.

Note that the set up here captures the Group Actions 3 and 4 in Sect. 3.2. For example, suppose for Group Action 3, we consider linear codes which are n/2-dimensional subspaces of $\mathbb {F}_q^n$. Then we have $m=n/2$, so $\mathrm {PRA} (3)$ for this action does not hold, as $3>(m^2+n^2)/(mn)=5/2$.

On the other hand, when $d\le (m^2+n^2)/(mn)$, such an attack may fail, simply because of the existence of a dominant orbit.

5.2 The General Linear Action on Tensors as a Pseudorandom Action Candidate

We have explained why the general linear action on tensors is a good candidate for the one-way assumption in Sect. 3. We now argue that, to the best of our knowledge, it is also a candidate for the pseudorandom assumption.

We have described the current status of average-case algorithms for 3-tensor isomorphism problem in Sect. 3.3. One may expect that, because of the relaxed requirement for the average-case setting as discussed in Sect. 5.1, the algorithms in [15, 59] may be accelerated. However, this is not the case, because these algorithms inherently enumerate all vectors in $\mathbb {F}_q^n$, or improve somewhat by using the birthday paradox.

We can also explain why the relaxed requirement for the average-case setting is still very difficult, by drawing experiences from computational group theory, because of the relation between $\mathrm {GLAT}$ and Group Action 5, which in turn is closely related to the group isomorphism problem as explained in Sect. 3.2. In group theory, it is known that the number of non-isomorphic p-groups of class 2 and exponent p of order $p^\ell$ is bounded as $p^{\frac{2}{27}\ell ^3+\varTheta (\ell ^2)}$ [12]. The relaxed average-case requirement in this case then asks for an algorithm that could test isomorphism for a subclass of such groups containing non-isomorphic groups as many as $p^{\frac{2}{27}\ell ^3+\varTheta (\ell ^2)}/{{\,\mathrm{\mathrm {poly}}\,}}(\ell , \log p)=p^{\frac{2}{27}\ell ^3+\varTheta (\ell ^2)}$. This is widely regarded as a formidable task in computational group theory: at present, we only know of a subclass of such groups with $p^{O(\ell ^2)}$ many non-isomorphic groups that allows for an efficient isomorphism test [58].

The supergroup attack seems not useful here. The group $G={{\,\mathrm{\mathrm {GL}}\,}}(\ell , \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(n, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})$ naturally lives in ${{\,\mathrm{\mathrm {GL}}\,}}(\ell n m, \mathbb {F})$. However, by Aschbacher’s classification of maximal subgroups of finite classical groups [2], there are few natural supergroups of G in ${{\,\mathrm{\mathrm {GL}}\,}}(\ell n m, \mathbb {F})$. The obvious ones include subgroups isomorphic to ${{\,\mathrm{\mathrm {GL}}\,}}(\ell n, \mathbb {F})\times {{\,\mathrm{\mathrm {GL}}\,}}(m, \mathbb {F})$, which is not useful because it has a dominant orbit (Definition 9).

The geometric attack seems not useful here either. The invariant ring here is trivial [31]⁶. For the orbit closure intersection problem, despite some recent exciting progress in [1, 19, 20, 29, 52], the current best algorithms for the corresponding orbit closure intersection problems still require exponential time.

Finally, for the most general isomorphism invariant attack, the celebrated paper of Hillar and Lim [49] is just titled “Most Tensor Problems Are NP-Hard.” This suggests that getting one easy-to-compute and useful isomorphism invariant for $\mathrm {GLAT}$ is already a challenging task. Here, useful means that the invariant does not lead to an equivalence relation with a dominant class in the sense of Definition 9.

The above discussions not only provide evidence for $\mathrm {GLAT}$ to be pseudorandom, but also highlight how this problem connects to various mathematical and computational disciplines. We believe that this could serve a further motivation for all these works in various fields.