A Critical Analysis of ISO 17825 (‘Testing Methods for the Mitigation of Non-invasive Attack Classes Against Cryptographic Modules’)

Abstract

The ISO standardisation of ‘Testing methods for the mitigation of non-invasive attack classes against cryptographic modules’ (ISO/IEC 17825:2016) specifies the use of the Test Vector Leakage Assessment (TVLA) framework as the sole measure to assess whether or not an implementation of (symmetric) cryptography is vulnerable to differential side-channel attacks. It is the only publicly available standard of this kind, and the first side-channel assessment regime to exclusively rely on a TVLA instantiation.

TVLA essentially specifies statistical leakage detection tests with the aim of removing the burden of having to test against an ever increasing number of attack vectors. It offers the tantalising prospect of ‘conformance testing’: if a device passes TVLA, then, one is led to hope, the device would be secure against all (first-order) differential side-channel attacks.

In this paper we provide a statistical assessment of the specific instantiation of TVLA in this standard. This task leads us to inquire whether (or not) it is possible to assess the side-channel security of a device via leakage detection (TVLA) only. We find a number of grave issues in the standard and its adaptation of the original TVLA guidelines. We propose some innovations on existing methodologies and finish by giving recommendations for best practice and the responsible reporting of outcomes.

Appendices

A Sample Size for the t-Test

We begin with a simple visual example that illustrates the concepts of $\alpha$ and $\beta$ values and their relationship to the sample size.

Consider the following two-sided hypothesis test for the mean of a Gaussian-distributed variable $A \sim \mathcal {N}(\mu ,\sigma )$, where $\mu$ and $\sigma$ are the (unknown) parameters:

$$\begin{aligned} H_0: \mu = \mu _0 \text { vs.}&\text { } H_{alt}: \mu \ne \mu _0. \end{aligned}$$

(3)

Note that, in the leakage detection setting, where one typically wishes to test for a non-zero difference in means between two Gaussian distributions $Y_1$ and $Y_2$, this can be achieved by defining $A = Y_1 - Y_2$ and (via the properties of the Gaussian distribution) performing the above test with $\mu _0 = 0$.

Suppose the alternative hypothesis is true and that $\mu = \mu _{alt}$. This is called a ‘specific alternative’⁸, in recognition of the fact that it is not usually possible to compute power for all the alternatives when $H_{alt}$ defines a set or range. In the leakage detection setting one typically chooses $\mu _{alt} > 0$ to be the smallest difference $|\mu _1 - \mu _2|$ that is considered of practical relevance; this is called the effect size. Without loss of generality, we suppose that $\mu _{alt} > \mu _0$.

Figure 5 illustrates the test procedure when the risk of a Type I error is set to $\alpha$ and the sample size is presumed large enough (typically $n>30$) that the distributions of the test statistic under the null and alternative hypotheses can be approximated by Gaussian distributions. The red areas together sum to $\alpha$; the blue area indicates the overlap of $H_0$ and $H_{alt}$ and corresponds to $\beta$ (the risk of a Type II error). The power of the test – that is, the probability of correctly rejecting the null hypothesis when the alternative in true – is then $1-\beta$, as depicted by the shaded area.

There are essentially three ways to raise the power of the test. One is to increase the effect size of interest which, as should be clear from Fig. 5, serves to push the distributions apart, thereby diminishing the overlap between them. Another is to increase $\alpha$ – that is, to make a trade-off between Type II and Type I errors – or (if appropriate) to perform a one-sided test, either of which has the effect (in this case) of shifting the critical value to the left so that the shaded region becomes larger. (In the leakage detection case the one-sided test is unlikely to be suitable as differences in either direction are equally important and neither can be ruled out a priori). The third way to increase the power is to increase the sample size for the experiment. This reduces the standard error on the sample means, which again pushes the alternative distribution of the test statistic further away from null (note from Fig. 5 that it features in the denominator of the distance).

Suppose you have an effect size in mind – based either on observations made during similar previous experiments, or on a subjective value judgement about how large an effect needs to be before it is practically relevant (e.g. the level of leakage which is deemed intolerable) – and you want your test to have a given confidence level $\alpha$ and power $1-\beta$. The relationship between confidence, power, effect size and sample size can then be used to derive the minimum sample size necessary to achieve this.

The details of the argumentation that now follows are specific to a two-tailed t-test, but the general procedure can be adapted to any test for which the distribution of the test statistic is known under the null and alternative hypotheses.

For the sake of simplicity (i.e. to avoid calculating effectively irrelevant degrees of freedom) we will assume that our test will in any case require the acquisition of more than 30 observations, so that the Gaussian approximations for the test statistics hold as in Fig. 5. Without loss of generality we also assume that the difference of means is positive (otherwise the sets can be easily swapped). Finally, we assume that we seek to populate both sets with equal numbers $n=|Y|/2$ of observed traces.

Theorem 1

Let $Y_1$ be a set of traces of size N/2 drawn via repeat sampling from a normal distribution $\mathcal {N}(\mu _1,\sigma _1^2)$ and $Y_2$ be a set of traces of size N/2 drawn via repeat sampling from a normal distribution $\mathcal {N}(\mu _2,\sigma _2^2)$. Then, in a two-tailed test for a difference between the sample means:

$$\begin{aligned} H_0 \text {: } \mu _1 = \mu _2 \text { vs. } H_{alt} \text {: } \mu _1 \ne \mu _2, \end{aligned}$$

(4)

in order to achieve significance level $\alpha$ and power $1-\beta$, the overall number of traces N needs to be chosen such that:

$$\begin{aligned} N&\ge 2\cdot \frac{(z_{\alpha /2}+z_\beta )^2\cdot ({\sigma _1}^2 + {\sigma _2}^2)}{(\mu _1-\mu _2)^2}. \end{aligned}$$

(5)

Fig. 5.

Figure showing the Type I and II error probabilities, $\alpha$ and $\beta$ as well as the effect size $\mu _{alt}-\mu _0$ for a specific alternative such that $\mu _{alt} > \mu _0$.

Note that Eq. 5 can be straightforwardly rearranged to alternatively compute any of the significance level, effect size or power in terms of the other three quantities.

B Results for Original TVLA-Recommended Threshold

Table 6. LHS: Power to achieve Cohen’s and Sawilowsky’s standardised effects under the TVLA significance criteria (which approximates to $\alpha = 0.00001$) and the standard level 3 ($N = 10,000$) and level 4 ($N = 100,000$) sample size criteria; RHS: Minimum effect sizes detectable for increasing power thresholds.

Table 7. Average (‘per-test’) and 1-minimal (‘overall’) power to detect observed and ‘tiny’ effect sizes under the level 3 and 4 criteria, and the sample size required to achieve balanced errors for a significance criterion of $\alpha = 0.00001$. (30 leak points in a trace set of length 1,400).

Fig. 6.

Different types of power and error to detect 30 true effects of size 0.04 in a trace set of length 1,400, as sample size increases, for an overall significance level of $\alpha = 0.00001$. (Based on 5,000 random draws from the multivariate test statistic distribution under the alternative hypothesis).

Table 8. Different types of power and error to detect 30 true effects of size 0.04 in a trace set of length 1,400, under the level 3 and level 4 sample size criteria and with an overall significance level of $\alpha = 0.00001$. (Based on 5,000 random draws from the multivariate test statistic distribution under the alternative hypothesis).

Fig. 7.

FWER and 1-minimal (‘overall’) power of the tests to detect effects of the ‘observed’ size 0.04 for various leakage scenarios as the trace length increases, under the level 3 and level 4 standard criteria with an overall significance level of $\alpha = 0.00001$.