Constraining multi-Higgs flavour models

Abstract

To study a flavour model with a non-minimal Higgs sector one must first define the symmetries of the fields; then identify what types of vacua exist and how they may break the symmetries; and finally determine whether the remnant symmetries are compatible with the experimental data. Here we address all these issues in the context of flavour models with any number of Higgs doublets. We stress the importance of analysing the Higgs vacuum expectation values that are pseudo-invariant under the generators of all subgroups. It is shown that the only way of obtaining a physical CKM mixing matrix and, simultaneously, non-degenerate and non-zero quark masses is requiring the vacuum expectation values of the Higgs fields to break completely the full flavour group, except possibly for some symmetry belonging to baryon number. The application of this technique to some illustrative examples, such as the flavour groups $\Delta (27)$, $A_4$ and $S_3$, is also presented.

1 Introduction

The origin of the flavour structure of the fermion masses and mixing in the standard model (SM) remains one of the unsolved puzzles in particle physics. Several approaches to this problem have been put forward, most of them based on the use of discrete [1–4] or continuous [5] flavour symmetries. In particular, mainly motivated by the measurement of several neutrino oscillation parameters, the use of discrete symmetries has recently become more popular (for recent reviews, see e.g. Refs. [6–9]).

In flavour model building, one commonly chooses a flavour symmetry group $K$ and then studies how fermion masses and mixing are constrained by this symmetry. Because of the SM gauge symmetry there is, of course, an additional global hypercharge symmetry $U(1)_Y$ in the Lagrangian. Thus, in such studies it is important to distinguish the chosen flavour group $K$ from the full flavour group, denoted henceforth by $G=K\times U(1)_Y$. After spontaneous symmetry breaking (SSB) the vacuum expectation values (vevs) of the Higgs fields break completely, in many instances, $K$ and $G$, leaving no residual flavour symmetry in the model. But, in some rather interesting cases, the vevs could fully break $K$ but not $G$, and thus some residual symmetry remains. Clearly, any accidental symmetry of the Lagrangian, such as $U(1)_B$, corresponding to baryon number, is not relevant for the analysis of the flavour sector neither any residual symmetry of $G$ which is contained in it.

In the literature, there are studies focusing on symmetry groups exclusively applied to the Higgs sector, either with two [10–12] or more than two doublets [13–15], as well as studies focusing either on flavour symmetry groups acting in the fermion sector with assumed vev alignments [6–9, 16, 17] or on the residual symmetries remaining on the fermion mass matrices [18–22]. Our aim is to provide a contribution towards the connection of all these subjects. In particular, we shall study in a systematic fashion the impact of a chosen flavour group on the Higgs potential and its vevs, on the Yukawa coupling matrices and thus on the residual physical properties of the mass matrices and the Cabibbo–Kobayashi–Maskawa (CKM) quark mixing matrix.

The introduction of additional scalars is a common feature to all flavour models, which brings with it a large number of possible model implementations for a given flavour group. Studying all these implementations, even for two or three Higgs doublet models, is not an easy task, specially for large flavour groups. We shall show that most of the unphysical scenarios can be identified just by analysing how scalars transform under the flavour group and how they break it. Through this analysis we shall prove that under very general conditions some choices of flavour groups are excluded for a given scalar content. Although we shall restrict our discussion to the quark sector of the theory, the results presented here can easily be extended to the lepton sector.

The paper is organised as follows. In Sect. 2, after briefly reviewing our notation and the experimental input, we analyse the invariance of the Yukawa coupling matrices under a full flavour group. Then we proceed to the formulation of a theorem which improves that in Ref. [18], by including also the constraints arising from the right-handed quark sector. A summary of the possible CKM mixing patterns and quark spectra is presented, according to the existence or not of a residual symmetry in the whole quark sector. Most importantly, by introducing the notion of pseudo-invariance of the vev under all subgroups of the flavour group, we address for the first time the problem of knowing whether such a residual symmetry is possible or not. This issue has not been addressed before in the literature, and is explained in detail in Sect. 3, where we illustrate the technique by applying it to the flavour groups $\Delta (27)$, $A_4$ and $S_3$, commonly found in the literature to explain fermion masses and mixing. Finally, our concluding remarks are given in Sect. 4.

2 A no-go theorem for models with $N$ Higgs doublets

2.1 Notation and experimental input

Let us consider a model with $N$ Higgs doublets, $\phi _k$, and three generations of left-handed quark doublets, $Q_L = (u_L, d_L)^T$, right-handed up-type quark singlets, $u_R$, and right-handed down-type quark singlets, $d_R$. The Yukawa Lagrangian is

$$\begin{aligned} -{\mathcal L}_Y = \bar{Q}_L\, \varGamma _k\, \phi _k\, d_R + \bar{Q}_L\, \Delta _k\, \tilde{\phi }_k\, u_R + \text {H.c.}, \end{aligned}$$

(1)

where a sum over the Higgs fields ($k=1 \dots N$) is implicit, $\varGamma _k$ and $\Delta _k$ are $3 \times 3$ matrices in the down- and up-type quark family space, respectively. The scalar fields can be combined into $\Phi = (\phi _1,\,\phi _2,\ldots ,\phi _N)^T$. After the SSB, the scalars acquire vevs, $\langle \Phi \rangle = (v_1, \, v_2, \ldots , v_N)^T$, and the Yukawa Lagrangian contains the terms

$$\begin{aligned} - \langle {\mathcal L}_Y \rangle&= \left( \bar{u}_L, \bar{d}_L \right) \, M_d\, \left( \begin{array}{l} 0\\ 1 \end{array} \right) \, d_R \nonumber \\&+\left( \bar{u}_L, \bar{d}_L \right) \, M_u\, \left( \begin{array}{c} 1\\ 0 \end{array} \right) \, u_R + \text {H.c.}, \end{aligned}$$

(2)

where

$$\begin{aligned} M_d = \varGamma _k\, v_k,\quad M_u = \Delta _k\, v_k^*, \end{aligned}$$

(3)

are the down-quark and up-quark mass matrices.

Using transformations that keep the $SU(2)_L$ doublet structure $\bar{Q}_L = \left( \bar{u}_L, \bar{d}_L \right)$, it is possible to diagonalise $M_d$ or $M_u$, but not both matrices. We denote by “left space”, the space of vectors $\bar{Q}_L$, which has three components in family space, one for each generation of quarks. So, $M_d$ and $M_u$ cannot be simultaneously diagonalised with left space transformations. To diagonalise them, we must treat $u_L$ and $d_L$ differently. Indeed, one can always find matrices $V_{uL}$, $V_{uR}$, $V_{dL}$, $V_{dR}$, such that

$$\begin{aligned} V_{dL}^\dagger \, M_d\, V_{dR}&= D_d = \text {diag}\,(m_d, m_s, m_b), \nonumber \\ V_{uL}^\dagger \, M_u\, V_{uR}&= D_u = \text {diag}\,(m_u, m_c, m_t). \end{aligned}$$

(4)

Often, it is convenient to work with the Hermitian matrices

$$\begin{aligned} H_d = M_d\, M_d^\dagger , \quad H_u = M_u\, M_u^\dagger , \end{aligned}$$

(5)

which live on the left space only and remain invariant under unitary redefinitions of $u_R$ and/or $d_R$. Using Eqs. (4) and (5), we find

$$\begin{aligned} H_d = V_{dL}\, D_d^2\, V_{dL}^\dagger , \quad H_u = V_{uL}\, D_u^2\, V_{uL}^\dagger . \end{aligned}$$

(6)

We know from experiment that the quark masses are non-vanishing and non-degenerate. As a result, the matrices $H_d$ and $H_u$ are invertible and have a non-degenerate spectrum. The CKM mixing matrix is given by

$$\begin{aligned} V = V_{uL}^\dagger V_{dL}. \end{aligned}$$

(7)

It is also known from experiment that $V$ must have three non-trivial angles and that the largest contributor to CP violation in the kaon and $B$ meson systems must be the CKM CP-violating phase, which is proportional to the basis-invariant observable,

$$\begin{aligned} J = \text {Det} \left( H_d H_u - H_u H_d\right) . \end{aligned}$$

(8)

Therefore, $V$ cannot be block-diagonal or any permutation of a block-diagonal matrix, it cannot be given by the identity or by some permutation matrix, and $J$ must differ from zero.

2.2 Yukawa coupling invariance under a full flavour group $G$

Let us consider a group $G$ with group elements $g$. The group can be defined by a set of generators $i$. For example, we may have two generators, $a$ and $b$, such that all group elements are obtained by successive products of $a$ and $b$. Then one chooses how the various fields transform under the group $G$. This means that for three quark generations one chooses $3 \times 3$ matrices representing this group, so that the quark fields transform under the generators of the group as

$$\begin{aligned}&\!\!\!Q^G_{L\alpha }= \left( \mathcal {G}_{Li}\right) _{\alpha \beta }Q_{L\beta }, \nonumber \\&\!\!\!d^G_{R\alpha }= \left( \mathcal {G}^{d}_{Ri}\right) _{\alpha \beta }d_{R\beta },\\&\!\!\!u^G_{R\alpha }= \left( \mathcal {G}^{u}_{Ri}\right) _{\alpha \beta }u_{R\beta },\nonumber \end{aligned}$$

(9)

where the index $i$ runs from $1$ to the number of group generators. Similarly, one chooses $N \times N$ matrices, such that the scalar fields transform as

$$\begin{aligned} \Phi ^G_{k}= \left( \mathcal {G}_{i}\right) _{kl}\Phi _{l}. \end{aligned}$$

(10)

These equations hold for matrices $\mathcal {G}_i$ representing the generators as well as for matrices representing any element of the group, i.e. $\mathcal {G}_g$. The explicit form of $\mathcal {G}_{Li}$, $\mathcal {G}^{u}_{Ri}$, $\mathcal {G}^{d}_{Ri}$ and $\mathcal {G}_{i}$ dictates which representations $D(Q_L)$, $D(u_R)$, $D(d_R)$ and $D(\Phi )$, respectively, we have chosen for the fields.

We now ask the Yukawa Lagrangian to be invariant under this group before SSB. Using Eq. (1), this implies the following relations for the couplings:

$$\begin{aligned} \mathcal {G}_{Lg}^\dagger \, \varGamma ^k\,\mathcal {G}_{Rg}^d \, \left( \mathcal {G}_g\right) _{kl} = \varGamma ^l, \nonumber \\ \mathcal {G}_{Lg}^\dagger \, \Delta ^k\,\mathcal {G}_{Rg}^u \, \left( \mathcal {G}_g^*\right) _{kl} = \Delta ^l\, , \end{aligned}$$

(11)

for all group elements $g$. Equation (11) is clearly valid for the identity element $g=e$. It is also valid for an element $g$ such that $\mathcal {G}_{Lg}=\mathcal {G}_{Rg}=e^{i\theta }{1}\mathrm{l}$ and $\mathcal {G}_g={1}\mathrm{l}$. The latter relations are obviously always verified since baryon number is conserved at or below the electroweak scale. We call such an element a trivial group element, the group it generates ($G_g$) a trivial group, and we denote this case by $G_g\subseteq U(1)_B$.

To decide what invariance may remain in the Yukawa Lagrangian after SSB (and, therefore, in the mass matrices), we must know how the vacuum transforms under the group elements. The ensuing analysis may not be straightforward, so we will start with a simple case, and generalise it in steps. For the moment, we make three simplifying assumptions:

• (i) we ignore the up-type quarks;

• (ii) we assume that $\Phi$ is in an irreducible representation (irrep) of $G$;

• (iii) we assume that there is a non-trivial group element, $g_1$, leaving the vacuum invariant.

Given assumption (ii), we conclude that either no scalar couples to the down-type quarks (which would lead to massless quarks) or else all scalars couple to down-type quarks. This conclusion is a particular case of the following more general assertion:

Proposition

For the down-type Yukawa terms in Eq. (1), each set of Higgs doublets $\phi _k$, comprising an irreducible representation of $G$, either couples to quark fields with linearly independent $\varGamma ^k$, or decouples completely with $\varGamma ^k=0$.

The same proposition holds for the up-type Yukawa terms and their coefficients $\Delta ^k$. The proof is given in Appendix A; its essential idea is that if some $\varGamma ^k$ were linearly dependent, then one would be able to identify doublets which do not couple to the down-quark fields and represent an invariant subspace under the action of $G$, this making the representation reducible. Thus, no matrix $\varGamma ^k$ vanishes and

$$\begin{aligned} \left( \mathcal {G}_{g_1} \right) _{kl}\, v_l = v_k, \end{aligned}$$

(12)

for some given group element $g_1$. The set of all elements $g_1$ satisfying Eq. (12) forms a subgroup of $G$, which we denote by $G_q$. The Lagrangian after SSB is thus left invariant by a residual symmetry $G_q$. Then, specializing in Eq. (11) to an element $g=g_1$ in $G_q$, and using Eqs. (3) and (12), one gets

$$\begin{aligned} \mathcal {G}_{L{g_1}}^\dagger M_d\ \mathcal {G}_{R{g_1}}^d=M_d. \end{aligned}$$

(13)

Under these assumptions, $M_d$ is symmetric under the group element $g_1$ and the subgroup generated by it.

If assumption (ii) remains valid but we substitute (iii) by the assumption that the element $g \in U(1)_B$ is trivial (in the newly defined sense), then we conclude that $M_d$ has no symmetry, i.e., the group $G$ has been completely broken by the vacuum. In this case, we call the residual symmetry trivial, i.e. $G_q\subseteq U(1)_B$.

We now turn to the possibility that $\Phi$ is in a reducible representation of $G$, thus negating our assumption (ii). First we assume that $\Phi$ breaks into the three vectors

$$\begin{aligned} \left( \phi _1, \ldots , \phi _r \right) , \ \ \ \left( \phi _{r+1}, \ldots , \phi _s \right) , \ \ \ \left( \phi _{s+1}, \ldots , \phi _N \right) , \end{aligned}$$

(14)

which transform as the irreducible representations $\varphi _1$, $\varphi _2$, and $\varphi _3$ of $G$, respectively. We can still write Eq. (11). The only novelty is that $\mathcal {G}_g$ is now a block-diagonal matrix, with one $r \times r$ block, one $(s-r) \times (s-r)$ block, and one $(N-s) \times (N-s)$ block. Said otherwise, we may write

$$\begin{aligned} \mathcal {G}_{Lg}^\dagger \, \varGamma ^k_{\varphi _1}, \mathcal {G}_{Rg}^d \, \left( \mathcal {G}_g^{\varphi _1} \right) _{kl}&= \varGamma ^l_{\varphi _1}, \ \ k,l = 1 \dots r, \end{aligned}$$

(15)

$$\begin{aligned} \mathcal {G}_{Lg}^\dagger \, \varGamma ^k_{\varphi _2}, \mathcal {G}_{Rg}^d \, \left( \mathcal {G}_g^{\varphi _2} \right) _{kl}&= \varGamma ^l_{\varphi _2}, \ \ k,l = r+1 \dots s, \end{aligned}$$

(16)

$$\begin{aligned} \mathcal {G}_{Lg}^\dagger \, \varGamma ^k_{\varphi _3}, \mathcal {G}_{Rg}^d \, \left( \mathcal {G}_g^{\varphi _3} \right) _{kl}&= \varGamma ^l_{\varphi _3}, \ \ k,l = s+1 \dots N, \end{aligned}$$

(17)

where we introduce a label $\varphi _j$ denoting the representation. We assume that the $\varphi _3$ representation is such that the fields $\left( \phi _{s+1}, \ldots , \phi _N \right)$ do not couple to the down-type quarks, i.e., $\varGamma ^k_{\varphi _3} = 0$. In this case,

$$\begin{aligned} M_d = \sum _{k=1}^N \varGamma ^k v_k = \sum _{k=1}^r \varGamma ^k_{\varphi _1} v_k^{\varphi _1} + \sum _{k=r+1}^s \varGamma ^k_{\varphi _2} v_k^{\varphi _2}. \end{aligned}$$

(18)

We further assume that the vevs of $\left( \phi _{1}, \ldots , \phi _r \right)$ and $\left( \phi _{r+1}, \ldots , \phi _s \right)$ are invariant under some subgroups $G_1$ and $G_2$, respectively. Thus,

$$\begin{aligned} \left( \mathcal {G}_{g_1}\right) _{kl}\, v_l^{\varphi _1} = v_k^{\varphi _1}, \quad \left( \mathcal {G}_{g_2}\right) _{kl}\, v_l^{\varphi _2} = v_k^{\varphi _2}, \end{aligned}$$

(19)

for $g_1\in G_1$ and $g_2 \in G_2$. The common residual symmetry is now $G_q=G_1\cap G_2$. If $G_q\subseteq U(1)_B$, then the residual symmetry is trivial. In contrast, if we have a non-trivial $G_q\not \subseteq U(1)_B$, we can pick $g\in G_q$ and using Eqs. (15) and (16), together with Eqs. (18) and (19), we obtain again Eq. (13). Notice that, in this example, $\varphi _3$ is irrelevant for $M_d$ but it may matter for $M_u$, should $\left( \phi _{s+1}, \ldots , \phi _N \right)$ couple to the up-type quarks.

We are now ready to generalise the impact of symmetries and to include the up-quark sector. The residual symmetry $G_q$ restricted to the down-quark sector is now renamed $G_d$, while $G_q$ is reserved for the residual symmetry of the quark sector as a whole. If $\Phi$ is in a reducible representation of $G$, it may be seen as a sum of irreducible representations $\varphi _i$ as

$$\begin{aligned} \Phi = \varphi _1 \oplus \varphi _2 \oplus \cdots \oplus \varphi _m \quad \text {with} \quad m \le N. \end{aligned}$$

(20)

Equation (11) can now be written as a set of equations for each $\varphi _j$,

$$\begin{aligned} \begin{aligned} \mathcal {G}_{L{g}}^\dagger \, \varGamma ^k_{\varphi _j}\,\mathcal {G}_{R{g}}^d \, \left( \mathcal {G}^{\varphi _j}_{g} \right) _{kl} = \varGamma ^l_{\varphi _j},\\ \mathcal {G}_{L{g}}^\dagger \, \Delta ^k_{\varphi _j}\,\mathcal {G}_{R{g}}^u \, \left( \mathcal {G}_{g}^{\varphi _j} \right) ^*_{kl} = \Delta ^l_{\varphi _j}, \end{aligned} \end{aligned}$$

(21)

where no sum is assumed in $\varphi _j$. The matrices $\varGamma ^k_{\varphi _j}$ and $\Delta ^k_{\varphi _j}$ denote the couplings associated with $\phi _k$ for a given irreducible representation $\varphi _j$. For example, if $\Phi$ is an irreducible quadruplet $(\phi _1,\phi _2,\phi _3,\phi _4)$ then $\varGamma ^k$, for $k=1 \dots 4$, denotes the Yukawa coupling associated with each entry of the scalar representation. If $\Phi$ is a reducible quadruplet, composed by two irreducible doublets $\varphi _1 = (\phi _1,\phi _2)$ and $\varphi _2 = (\phi _3,\phi _4)$, then the coupling $\varGamma _{\varphi _i}^k$ associates two couplings to each irreducible representation, i.e. $k=1,2$. One denotes the vev of the irreducible representations $\varphi _j$ as $v^{\varphi _j}$. When the vev breaks the group $G$, the Lagrangian may still remain invariant under the action of some elements of $G$.

Let us denote by $G_d$ and $G_u$ the subgroups of $G$ that are left invariant by the vevs of all Higgs fields coupling to down-type and up-type quarks, respectively. Some fields may interact with both sectors. The relevant residual symmetry of the quark sector is $G_q=G_d\cap G_u$. The fields that do not couple to quarks are irrelevant to our discussion. There are several possibilities:

• (1) If $G_q\subseteq U(1)_B$, there is no non-trivial symmetry left in the quark sector as a whole. More specifically,

  • (a) If $G_{d,u} \subseteq U(1)_B$, then there is no non-trivial symmetry left in either (up- or down-) quark sector;

  • (b) If $G_d \not \subseteq U(1)_B$ and $G_u\subseteq U(1)_B$, then there is some non-trivial symmetry left in $M_d$, but not in $M_u$;

  • (c) If $G_d \subseteq U(1)_B$ and $G_u \not \subseteq U(1)_B$, then there is some non-trivial symmetry left in $M_u$, but not in $M_d$;

  • (d) If $G_d \not \subseteq U(1)_B$ and $G_u \not \subseteq U(1)_B$, then there is some non-trivial symmetry left in $M_d$, some non-trivial symmetry left in $M_u$, but there is no non-trivial symmetry left in both $M_d$ and $M_u$;

• (2) If $G_q\not \subseteq U(1)_B$, then there is some non-trivial symmetry left in the quark sector as a whole.

The characteristic in common to the cases in (1) is that $\langle \Phi \rangle$ breaks the symmetry completely, modulo $U(1)_B$. In contrast, case (2) leaves a common non-trivial symmetry in the whole quark sector. In the latter case, for $g \in G_q$, from Eqs. (21) we find

$$\begin{aligned} \mathcal {G}_{L{g}}^\dagger \, M_d\ \mathcal {G}_{R{g}}^d \, = M_d , \quad \mathcal {G}_{L{g}}^\dagger \, M_u\ \mathcal {G}_{R{g}}^u \, = M_u . \end{aligned}$$

(22)

with $M_d = \varGamma ^k_{\varphi _j} v^{\varphi _j}_k$ and $M_u = \Delta ^k_{\varphi _j} v_k^{\varphi _j\, *}$. Equation (22) generalises Eq. (13) to the case of reducible representations and is the main result of this section. Thus, when using Eq. (22), as we will do in the theorem to be proved in the next section, it does not matter whether the representation of the group $G$ is reducible or irreducible.

2.3 Theorem and proof

Let us consider a model with quarks and scalar fields transforming under some set of representations of a full flavour group $G$. We denote the representation space where $G$ acts as the flavour space, i.e., the horizontal space of replicated multiplets of quark and scalar fields with the same gauge quantum numbers. The following theorem strongly constrains the viable models.

Theorem

(No-Go) Given a group $G$ acting on the flavour space, the only way to obtain a non-block-diagonal CKM mixing matrix and, simultaneously, non-degenerate and non-zero quark masses, is that $\left<\Phi \right>$ breaks completely the group $G$, except possibly for some symmetry belonging to baryon number.

Proof

Suppose that there is a residual symmetry group $G_q$ which is left invariant by the vev $\left<\Phi \right>$. Let us denote a generic element of $G_q$ as $g_1$, and then Eq. (12) is assumed.¹ We want to show that either $g_1=e$, i.e., the trivial element in $G$, or $g_1$ acts on the flavour space as a member of the baryon number symmetry.

Equation (12) implies that the Lagrangian after SSB is invariant under $g_1$. Using the same rationale that led to Eq. (22), we conclude that

$$\begin{aligned} \mathcal {G}_{L{g_1}}^\dagger M_d\ \mathcal {G}_{R{g_1}}^d = M_d, \quad \mathcal {G}_{L{g_1}}^\dagger M_u\ \mathcal {G}_{R{g_1}}^u = M_u. \end{aligned}$$

(23)

Since we shall be interested in the left sector, where the CKM mixing arises, we may turn to the Hermitian combinations in Eq. (5), which verify

$$\begin{aligned} \mathcal {G}_{L{g_1}}^\dagger H_d\ \mathcal {G}_{L{g_1}} = H_d, \quad \mathcal {G}_{L{g_1}}^\dagger H_u\ \mathcal {G}_{L{g_1}} = H_u. \end{aligned}$$

(24)

The next step is to use the following result (proved in Appendix B): if the generator $\mathcal {G}_{Lg_1}$ given in Eq. (24) is not proportional to the identity matrix, then the CKM matrix is block-diagonal, in contradiction with experiment. Note that we include any permutation of a block-diagonal matrix, as well as the unit mixing matrix, $V={1}\mathrm{l}$, in the category of block-diagonal matrices. We then conclude that

$$\begin{aligned} \mathcal {G}_{L{g_1}} = e^{i \theta }\, {1}\mathrm{l}, \end{aligned}$$

(25)

so that Eq. (24) place no restriction on $H_{d,u}$, and one has a potentially viable CKM matrix.

Returning now to Eq. (23), it follows that

$$\begin{aligned} \mathcal {G}_{R{g_1}}^d = \mathcal {G}_{R{g_1}}^u = \mathcal {G}_{L{g_1}}, \end{aligned}$$

(26)

because $M_u$ and $M_d$ are invertible. Applying this to Eq. (11), we find

$$\begin{aligned} \varGamma ^k\, \left( \mathcal {G}_{g_1} -{1}\mathrm{l}\right) _{kl}=\varvec{0}, \quad \Delta ^k\, {\left( \mathcal {G}_{g_1}-{1}\mathrm{l}\right) ^*}_{kl}=\varvec{0} , \end{aligned}$$

(27)

for $l=1,\ldots ,N$. This implies that

$$\begin{aligned} \mathcal {G}_{g_1}= {1}\mathrm{l}, \end{aligned}$$

(28)

within each space where the set of Yukawa matrices $\{\varGamma ^k\}$ (or, analogously, $\{\Delta ^k\}$) is linearly independent. Since we are confined to Higgs doublets that couple to quarks—down-type, up-type or both—the proposition in Sect. 2.2 implies that one or both of the sets $\{\varGamma ^k\}$ and $\{\Delta ^k\}$ are composed of non-zero and linearly independent matrices for each irreducible sector of $\Phi$. Hence,

$$\begin{aligned} \mathcal {G}_{g_1} = {1}\mathrm{l}_{n \times n}, \end{aligned}$$

(29)

in the whole space of Higgs doublets that couple to quarks.

Finally, we can analyse Eqs. (25), (26) and (29) jointly. If $\Phi$ is in a faithful representation, then $g_1=e$ and no residual symmetry is present. In this case, $e^{i\theta }=1$ in Eq. (25). If $g_1\ne e$ ($e^{i\theta }\ne 1$), then $\Phi$ is unfaithful and some residual symmetry will be present in the final Lagrangian, without constraining the mixing. This residual symmetry should be specifically represented by Eqs. (25), (26) and (29), which is just part of the baryon number conservation. This completes the proof of the theorem, which generalises that in Ref. [18], by including the constraints on the right-handed quark sector. Without the latter constraints, theories leading to unphysical massless quarks would not be precluded.² $\square$

At this point two remarks are in order. First, any SM-like Lagrangian, as the one in Eq. (1), exhibits automatic conservation of the baryon number $U(1)_B$, independently of additional gauge or flavour symmetries. Such accidental symmetry imposes no constraints on masses and mixing, and remains conserved after electroweak symmetry breaking. The proof above shows that a non-block-diagonal $V_{CKM}$ is compatible with some residual symmetry inside $G$ only if the latter is a subgroup belonging to $U(1)_B$. Second, we should stress that Eq. (29) applies only to the Higgs doublets that couple to quarks. The Higgs doublets that appear solely in the scalar potential are important when analysing possible vacuum alignments, but they are irrelevant for the statement of the theorem.

Notice that no information of the scalar potential and vevs has been used in this proof. That is, the theorem constrains residual symmetries and can be applied to virtually any model. Nevertheless, our interest in this article lies on NHDM, and, as we will show in Sect. 3, the explicit transformation properties of the Higgs scalars, their vevs, and their relation to the subgroup chains are crucial, in particular, for the application of the theorem to specific symmetries of complete models of scalars and fermions.

2.4 Some illustrative examples

The theorem proved in the previous section is completely general; yet it is useful to study in more detail several particular cases that one may come across. We split them into four classes depending on whether the representations $D(Q_L)$ and $D(\Phi )$ of $G$, acting, respectively, on the left quark space $Q_L$ and the scalar space $\Phi$, are faithful or not. We denote the kernel of the representation of $D(Q_L)$ by $\ker D(Q_L)$ and write $\ker D(Q_L)=\{e\}$ if the representation is faithful. Otherwise $\ker D(Q_L)$ is the subgroup of $G$ which is mapped to the identity by the representation of $Q_L$.

• (i) $Q_L$ and $\Phi$ faithful: $\ker D(Q_L) =\ker D(\Phi )=\{e\}$ A direct application of the theorem implies that if $\left<\Phi \right>$ does not break $G$ completely, then there is a residual symmetry in the fermion sector leading to a block-diagonal mixing.

• (ii) $Q_L$ unfaithful and $\Phi$ faithful: $\ker D(Q_L)\ne \{e\}$ and $\ker D(\Phi )=\{e\}$ The full symmetry in the left sector is smaller than the one in the scalar sector and, in principle, we could have the proper subgroup $\ker D(Q_L)$ of $G$ (or smaller) unbroken by $\left<\Phi \right>$. However, Eq. (29) should hold and $\left<\Phi \right>$ should break $G$ completely. As an example, let us take $G=A_4$ with $Q_L\sim (\varvec{1},\varvec{1}',\varvec{1}'')$ and $\Phi \sim d_R\sim u_R\sim \varvec{3}$. If $\left<\Phi \right>\sim (1,0,0)$, the $\mathbb {Z}_2$ subgroup generated by $g_1=\mathrm {diag}\,(1,-1,-1)$ is conserved and is contained in $\ker D(Q_L)$. The representation $\mathcal {G}_{Lg_1}={1}\mathrm{l}_3$ satisfies Eq. (25), but Eqs. (26) and (29) are not valid as $\mathcal {G}_{g_1}=\mathcal {G}_{Rg_1}^d=\mathcal {G}_{Rg_1}^u=\mathrm {diag}\,(1,-1,-1)$. Therefore, we end up with a non-trivial residual symmetry, but the invertibility of $M_u$ and $M_d$ assumed in Eq. (26) is lost.

• (iii) $Q_L$ faithful and $\Phi$ unfaithful: $\ker D(Q_L)=\{e\}$ and $\ker D(\Phi ) \ne \{e\}$ This case is automatically excluded unless $\ker D(\Phi )$ acts like baryon number on quarks. The full symmetry of the potential is $G/\ker D(\Phi )$ and $\left<\Phi \right>$ can never break $\ker D(\Phi )$. As an example, let us take $G=A_4$ with $(\phi _1,\phi _2,\phi _3)\sim (\varvec{1},\varvec{1}',\varvec{1}'')$ and $Q_L\sim d_R\sim u_R\sim \varvec{3}$. If all $\phi _k$ get non-zero vevs, they only break $A_4$ to $\mathbb {Z}_2\times \mathbb {Z}_2$, which corresponds to the kernel of the representation of $\Phi$, and is generated by $\mathrm {diag}\,(1,-1,-1)$ and $\mathrm {diag}\,(-1,-1,1)$. This subgroup remains in the $Q_L$ quark sector and the CKM matrix would be trivial.

• (iv) $Q_L$ and $\Phi$ unfaithful: $\ker D(Q_L) \ne \{e\}$ and $\ker D(\Phi )$ $\ne \{e\}$ This case can be discarded given the Yukawa structure in Eq. (1) and the assumption that at least one of the representations for $Q_L,\Phi ,d_R,u_R$ is a faithful irrep or contains a faithful irrep (i.e. $G$, and not a smaller group, is the full flavour symmetry); see also Appendix C.

During the proof and consequent remarks of the theorem, nothing has been said about the reducibility or irreducibility of $\Phi$. If $\Phi$ is an irreducible representation of the full flavour group $G$, then the same field components $\phi _i$ couple to the up and down sectors. In the case that $\Phi$ is a reducible representation of $G$, the components $\phi _i$ can be arranged into irreducible multiplets just like in Eq. (20). In this case, there are many ways of breaking the flavour group $G$ completely in order to allow for models with non-zero and non-degenerate masses with viable $V_{CKM}$ (see end of Sect. 2.2 for the different possibilities).

We shall comment on case (1d) of Sect. 2.2, which is the most predictive and hence one of the most common approaches in model building, since each quark sector has its own non-trivial residual symmetry while the full flavour group $G$ is completely broken. This will lead, in general, to strong constraints to the mixing angles and the CP phase. For instance, in Ref. [23], the authors studied which would be the best residual symmetries in order to accommodate the experimental CKM mixing matrix. This was done in the context of the von Dyck flavour groups [21, 22]. It is shown that the flavour groups $D_N$ and $\Delta (6N^2)$, with $N$ an integer multiple of 7, are capable of determining the mixing angle $\theta _{12}$ close to its experimental value, if we fix $\theta _{23}=\theta _{13}=0$. Non-zero values for all three mixing angles require, on the other hand, infinite von Dyck groups.

We summarise in Table 1 the different possibilities for the CKM mixing angles and mass spectra depending on the existence (or not) of a residual symmetry in the whole theory. In the presence of a residual symmetry, not contained in $U(1)_B$, we can either have at most three non-zero mixing angles with some massless quarks, as in example (ii) above, or have massive quarks with at most one non-zero mixing angle, as in example (iii).

Table 1 Possible CKM mixing patterns and quark spectra depending on the presence of global residual symmetries, modulo $U(1)_B$. The Yes/No distinguishes if one/two quarks of the same type are massless

3 Application of the theorem to flavour models

The no-go theorem previously demonstrated strongly constrains viable flavour models based on a full flavour symmetry $G$ of the Lagrangian. However, it is crucial for its application that one can determine the symmetry properties of the vevs. In this section, we shall perform this task by introducing the notion of pseudo-invariance of the vevs under subgroups of the original group. We illustrate the technique by applying it to some examples of flavour groups commonly found in the literature to explain fermion masses and mixing.

Before proceeding with the examples, it is worth recalling the distinction between what is meant by “full flavour group” and by simply “flavour group”. As said before, in model building, we may add symmetries that act only on the flavour space of the model. Those symmetries are usually taken to be subgroups of $SU(N)$ (in particular $SU(3)$, when we are dealing with three fermion generations). We refer to these symmetries as flavour symmetries $K$. However, the full flavour symmetry group $G$ is, in general, larger since it contains the global hypercharge transformation, i.e. $G=K\times U(1)_Y$.³ The accidental baryon symmetry $U(1)_B$ is not included in the full flavour symmetry $G$ because the Higgs fields do not transform under it, but it might happen that $G$ intersects $U(1)_B$. We further assume that there is no additional accidental symmetry in the Yukawa Lagrangian (1), apart from $G$ and $U(1)_B$ (see Refs. [24, 25] for methods for detecting them).

The additional $U(1)_Y$ in the full flavour symmetry is of extreme importance. As we shall see, one may break completely the flavour symmetry $K$ but still leave a non-trivial residual symmetry in $G$. To better understand this scenario, we introduce the notion of pseudo-invariant vacuum. In our discussion we have used the invariance of the vacuum, i.e. Eq. (12), in order to constrain the possible viable models. This vacuum invariance has been defined in the full flavour group $G$. Looking only to the flavour group $K$, we may extend the notion of invariance to pseudo-invariance of an irrep through the condition

$$\begin{aligned} \left( \mathcal {G}_{\tilde{g}_1} \right) _{kl}\, v_l = e^{i\alpha }v_k\quad \text {with}\quad \tilde{g}_1\in K. \end{aligned}$$

(30)

The group element $\tilde{g}_1$ changes the vacuum by a global phase transformation. In general, these global phase transformations are not contained in $K$, but they are contained in $G$ due to the $U(1)_Y$ global group. This means that, for a given irrep, any pseudo-invariant vacuum in $K$ can be written as an invariant one in $G$. Therefore, instead of dealing with the full flavour group $G$, we may restrict ourselves to the flavour group $K$ and its non-trivial pseudo-invariant vevs. Note that if $\Phi$ is composed of more than one irrep $\varphi _j$, as in Eq. (20), a pseudo-invariant vev where

$$\begin{aligned} \left( \mathcal {G}_{\tilde{g}_1} \right) _{kl}\, v_l^{\varphi _j} = e^{i\alpha _j}v_k^{\varphi _j},\quad \text {with}\quad \tilde{g}_1\in K, \end{aligned}$$

(31)

is equivalent to an invariant vev of $G$ only if all vevs in each irrep transform by the same global phase, i.e., $\alpha _j=\alpha$. One consequence immediately follows: the presence of an invariant Higgs doublet, singlet with respect to $K$, may be relevant. For example, consider down-type quarks interacting only with one $\mathbf {n}$-dimensional irrep $\varphi _1=(\phi _1,\ldots ,\phi _n)$ whose vev is pseudo-invariant by an element $\tilde{g}_1 \ne e$ with $\alpha _1\ne 0$. The mass matrix $M_d$ will possess a non-trivial residual symmetry. Now, suppose we add a trivial irrep $\varphi _2=\phi _{n+1}\sim \mathbf {1}$ that also interacts with down-type quarks. Any vev for $\varphi _2$ will be strictly invariant by $K$, with $\alpha _2=0$ in Eq. (31). Hence, $M_d$ will no longer possess the previous residual symmetry.

We remark that Eq. (31) (and consequently Eq. (30)) is merely an eigenvalue equation. Therefore, from a simple group-theoretical method, and without analysing the scalar potential, we can extract a set of vacuum alignments that will be automatically excluded by the theorem, namely those corresponding to the eigenvectors of $\mathcal {G}_{\tilde{g}_1}$. Notice, however, that the vev alignments obtained through this procedure may not be global minima of the scalar potential. Yet, this is a straightforward group-theoretical check in the spirit familiar to model building. One could instead follow a geometrical method [14] to find first the global minima of the scalar potential. In the latter case, if all the minima preserve some subgroup of the initial symmetry, the theorem applies directly without the need of solving the eigenvalue equation (31). The drawback is that the geometrical approach is not universal, and it is not guaranteed that it could easily be applied to more complicated Higgs sectors, for example to some high symmetry groups in 4HDM.

Next we shall present three examples of flavour groups in the context of 3HDM, two of them with $\Phi$ being in a faithful triplet representation, for the $\Delta (27)$ and $A_4$ flavour groups, and one example with $\Phi$ in a reducible triplet representation, for the $S_3$ flavour group.

Fig. 1

Diagram of the subgroups of $\Delta (27)$ and their generators, based on Ref. [27]. Invariant subgroups are encased in diamond shaped boxes; others are in circles

3.1 The flavour group $\Delta (27)$ in 3HDM

The flavour group $\Delta (27)$ [26] is a subgroup of $SU(3)$ and can be viewed as the group defined by two generators $a$ and $b$, with presentation

$$\begin{aligned} a^3 = b^3 = (a b)^3 = 1. \end{aligned}$$

(32)

We use the specific three-dimensional implementation [13]:

$$\begin{aligned} a = \left( \begin{array}{l@{\quad }l@{\quad }l} 1 &{} 0 &{} 0\\ 0 &{} \omega &{} 0\\ 0 &{} 0 &{} \omega ^2 \end{array} \right) , \ \ \ b = \left( \begin{array}{l@{\quad }l@{\quad }l} 0 &{} 1 &{} 0\\ 0 &{} 0 &{} 1\\ 1 &{} 0 &{} 0 \end{array} \right) , \end{aligned}$$

(33)

where $\omega = e^{2 i \pi /3}$. The group has 27 elements divided into 11 irreducible representations: nine singlets $\mathbf {1}_{(i,j)}$, with $i,j = 0,1,2$; and two triplets, $\mathbf {3}_{(0,1)}$ and $\mathbf {3}_{(0,2)}$. There are two elements of particular interest. One is

$$\begin{aligned} d \equiv b^2\, a\, b = \left( \begin{array}{l@{\quad }l@{\quad }l} \omega ^2 &{} 0 &{} 0\\ 0 &{} 1 &{} 0\\ 0 &{} 0 &{} \omega \end{array} \right) . \end{aligned}$$

(34)

Indeed, since $\Delta (27)$ is isomorphic to $Z_3 \times Z_3 \rtimes Z_3$ we can view the first two $Z_3$ as generated by $a$ and $d$ (they obviously commute), while the third one is generated by $b$. The second element of interest is

$$\begin{aligned} h \equiv a^2\, b\, a\, b^2 = \omega \, \left( \begin{array}{l@{\quad }l@{\quad }l} 1 &{} 0 &{} 0\\ 0 &{} 1 &{} 0\\ 0 &{} 0 &{} 1 \end{array} \right) . \end{aligned}$$

(35)

The $Z_3$ group generated by $h$ is the only $Z_3$ invariant subgroup of $\Delta (27)$. A summary of the subgroups and their generators is presented in Fig. 1, based on Ref. [27]. The eigenvectors $(v_1, v_2, v_3)$ associated to the generators of the $Z_3 \times Z_3$ invariant subgroups of $\Delta (27)$ are given by

$$\begin{aligned} a^2b:&\ (1, 1, \omega ),\ \ (1, \omega , 1),\ \ (\omega , 1, 1);\nonumber \\ ab:&\ (1, 1, \omega ^2),\ \ (1, \omega ^2, 1),\ \ (\omega ^2, 1, 1);\nonumber \\ a:&\ (0, 0, 1),\ \ (0,1, 0),\ \ (1, 0, 0);\nonumber \\ b:&\ (1, 1, 1),\ \ (1,\omega , \omega ^2),\ \ (1,\omega ^2, \omega ). \end{aligned}$$

(36)

Any of the vacuum alignments presented above automatically leads to a non-trivial residual symmetry in the final Lagrangian. Actually, because all elements of the triplet irrep of $\Delta (27)$ have eigenvalues $1,\, \omega$ or $\omega ^2$, any pseudo-invariant vacuum alignment can be seen, with the help of $h$, as an invariant one of another element of $\Delta (27)$. Notice also that $h$ leaves pseudo-invariant any type of vacuum alignment. However, it is not of real interest since $h \in \mathbb {Z}_3$, i.e. is the center of $SU(3)$. As already pointed out, these transformations are included in the $U(1)_Y$ part of full flavour group $G$. Therefore, we just need to look at the $\Delta (27)/\mathbb {Z}_3$ group. In this case, some alignments in Eq. (36) will break the flavour group, but the full flavour group will still be broken to a non-trivial subgroup.

The group-theoretical method presented above allows us to exclude several alignments without the need of dealing with the details of the scalar potential. However, this method tell us nothing about the global minima of the potential. Minimizing the general scalar potential for 3HDM is a very hard task, and until now no good method has been developed. When the potential exhibits some symmetry, the geometrical method [14] can give us the answer (specially for large symmetry groups). This method has been used in the study of $\Delta (27)$ in 3HDM, where four classes of global minima are found [28], corresponding precisely to the eigenvectors given in Eq. (36).⁴ Therefore, for the flavour group $\Delta (27)$ all the global minima of the potential are excluded by the theorem. Said otherwise, one cannot construct a viable model based on $\Delta (27)$ where $\Phi$ is in a faithful (triplet) irrep.

Fig. 2

Diagram of the subgroups of $A_4$ and their generators, based on Ref. [27]. Invariant subgroups are encased in diamond shaped boxes; others are in circles

3.2 The flavour group $A_4$ in 3HDM

The flavour group $A_4$ can be viewed as the group defined by two generators $s$ and $t$, with presentation

$$\begin{aligned} s^2=t^3=(st)^3=1. \end{aligned}$$

(37)

The group $A_4$ has one non-trivial invariant subgroup; $Z_2\times Z_2$, generated by $s$ and $tst^2$. It also has four subgroups $Z_3$, generated by $t$, $tst$, $st$ and $st^2$, and three subgroups $Z_2$ generated by $s$, $t^2st$ and $tst^2$. A summary of the subgroups and their generators is shown in Fig. 2, based on Ref. [27].

A possible matrix representation for the three-dimensional irrep is

$$\begin{aligned} s= \begin{pmatrix} 1&{}\quad 0&{}\quad 0\\ 0&{}\quad -1&{}\quad 0\\ 0&{}\quad 0&{}\quad -1 \end{pmatrix},\quad t= \begin{pmatrix} 0&{}\quad 1&{}\quad 0\\ 0&{}\quad 0&{}\quad 1\\ 1&{}\quad 0&{}\quad 0 \end{pmatrix}. \end{aligned}$$

(38)

We can use the group-theoretical method in order to find the set of alignments forbidden by the theorem. In this case, the eigenvectors $(v_1, v_2, v_3)$ associated to the generators of the subgroups are

$$\begin{aligned} t:&\quad (1,1,1),\ \ (1,\omega ,\omega ^2),\ \ (1,\omega ^2,\omega );\nonumber \\ tst:&\quad (-1,1,1),\ \ (-1,\omega ,\omega ^2),\ \ (-1,\omega ^2,\omega );\nonumber \\ st:&\quad (1,-1,1),\ \ (1,-\omega ,\omega ^2),\ \ (1,-\omega ^2,\omega );\nonumber \\ st^2:&\quad (1,1,-1),\ \ (1,\omega ,-\omega ^2),\ \ (1,\omega ^2,-\omega );\nonumber \\ s:&\quad (1,0,0),\ \ (0,v_2,v_3);\nonumber \\ t^2st:&\quad (0,1,0),\ \ (v_1,0,v_3);\nonumber \\ tst^2:&\quad (0,0,1),\ \ (v_1,v_2,0), \end{aligned}$$

(39)

where $v_1, v_2$ and $v_3$ are complex parameters. Using the geometrical method, the global minima of the $A_4$ symmetric scalar potential were found to be [14]

$$\begin{aligned} (1,1,1),\, (1,0,0),\,(\pm 1,e^{i \pi /3},e^{-i \pi /3}), \,(1,e^{i\alpha },0), \end{aligned}$$

(40)

up to permutations and global phase transformations. All these global minima are included in the list of alignments given in Eq. (39). Indeed, the first two are obviously there; the third vev alignment is $(\pm 1,e^{i \pi /3},e^{-i \pi /3})$ $=-(\mp 1,\omega ^2,\omega )$ and is in the list as well. Finally, the last minimum is a particular case when $|v_1|=|v_2|$. In summary, each $A_4$ global minimum leaves invariant a non-trivial subgroup of the full flavour symmetry $G=A_4\times U(1)_Y$ and is thus excluded by the no-go theorem. That is, one cannot construct a viable model based on $A_4$ where $\Phi$ is in a faithful (triplet) irrep [16].

It is interesting to consider an example similar to the one we included in point (ii) of Sect. 2.4, where the representations are $Q_L\sim (\varvec{1},\varvec{1}',\varvec{1}'')$ and $\Phi \sim d_R\sim u_R\sim \varvec{3}$. Now we take $\left<\Phi \right> \sim (1, e^{i \alpha }, 0)$. This leaves invariant a residual subgroup with generators $\mathcal {G}_L = \text {diag}(1,1,1)$ and $\mathcal {G}_{R}^{u} = \mathcal {G}_{R}^{d} = \mathcal {G} = \text {diag}(1,1,-1)$. The mass matrices for the up- and down-quarks may be written as

$$\begin{aligned} \left( \begin{array}{l@{\quad }l@{\quad }l} a &{} a\, e^{i\alpha } &{} 0\\ b &{} b\, \omega \, e^{i\alpha } &{} 0\\ c &{} c\, \omega ^2\, e^{i\alpha } &{} 0 \end{array} \right) . \end{aligned}$$

(41)

This implies that, after up- and down-quark mass matrix diagonalisation, there remain three mixing angles and three non-degenerate quark masses. However, these matrices are not invertible, which signals the presence of massless quarks in both up and down sectors, in contradiction with the experiment. If we had looked exclusively at the left-handed quark sector, as in Ref. [18], we would have accepted this model as a viable one. Although the symmetry has been completely broken in the left sector, the full Lagrangian has an unbroken symmetry in the right-handed quark sector. It is precisely here where the residual symmetry acts non-trivially and the no-go theorem applies.

Fig. 3

Diagram of the subgroups of $S_3$ and their generators, based on Ref. [27]. Invariant subgroups are encased in diamond shaped boxes; others are in circles

3.3 The flavour group $S_3$ in 3HDM

Since the group $S_3$ is the smallest non-Abelian discrete symmetry, it has been widely used for flavour physics in the quark and lepton sectors (for recent works see e.g. Refs. [29, 30], and references therein). The flavour group $S_3$ can be viewed as the group defined by two generators, $a$ and $b$, with presentation

$$\begin{aligned} a^3=b^2=(ab)^2=1. \end{aligned}$$

(42)

It has the non-trivial invariant subgroup $Z_3$, which is generated by $a$, and three subgroups $Z_2$, generated by $b$, $ab$ and $a^2b$. We summarise in Fig. 3 these subgroups and their generators [27].

The $S_3$ flavour group has no three-dimensional irrep. Therefore, any 3HDM with a scalar potential invariant under this group, with a faithful representation, must be built from a two-dimensional irrep $\mathbf {2}$ and one of the one-dimensional irreps, $\mathbf {1}$ or $\mathbf {1}^\prime$. The representation of the generators for these irreps can be chosen as

$$\begin{aligned}&\!\mathbf {2}:\quad a=\begin{pmatrix} \omega &{}\quad 0\\ 0&{}\quad \omega ^2 \end{pmatrix},\, b= \begin{pmatrix} 0&{}\quad 1\\ 1&{}\quad 0 \end{pmatrix},\nonumber \\&\quad \!\mathbf {1}:\quad a=b=1,\\&\quad \!\mathbf {1}^\prime :\quad a=-b=1.\nonumber \end{aligned}$$

(43)

When building an $S_3$ flavour model we have to decide whether our reducible three-dimensional representation is $\mathbf {2}+\mathbf {1}$ or $\mathbf {2}+\mathbf {1}^\prime$. We may represent them in the compact form

$$\begin{aligned} \mathbf {3}_{\pm }:\quad a= \begin{pmatrix} \omega &{}\quad 0&{}\quad 0\\ 0&{}\quad \omega ^2&{}\quad 0\\ 0&{}\quad 0&{}\quad 1 \end{pmatrix},\quad b_{\pm }= \begin{pmatrix} 0&{}\quad 1&{}\quad 0\\ 1&{}\quad 0&{}\quad 0\\ 0&{}\quad 0&{}\quad \pm 1 \end{pmatrix}, \end{aligned}$$

(44)

with the plus and minus signs assigned to $\mathbf {2}+\mathbf {1}$ and $\mathbf {2}+\mathbf {1}^\prime$, respectively. Following the group-theoretical method, we can find the set of vev alignments forbidden by the theorem. In this case, the eigenvectors associated to the reducible generators of the subgroups will be denoted by $((v_1,v_2),u)$, in order to distinguish the doublet from the singlet irrep. The corresponding eigenvectors for the $\mathbf {\mathbf {3}}_{\pm }$ are

$$\begin{aligned} a:&\ ((1,0),0),\ ((0,1),0),\ ((0,0),1) ;\nonumber \\ b_+:&\ ((v,v),u),\ ((-1,1),0) ;\nonumber \\ b_-:&\ ((1,1),0),\ ((-v,v),u) ;\nonumber \\ ab_+:&\ ((v\,\omega ,v),u),\ ((-\omega ,1),0) ;\nonumber \\ ab_-:&\ ((\omega ,1),0),\ ((-v\,\omega ,v),u) ;\nonumber \\ a^2b_+:&\ ((v\,\omega ^2,v),u),\ ((1,-1),0) ;\nonumber \\ a^2b_-:&\ ((\omega ^2,1),0),\ ((-v\,\omega ^2,v),u) . \end{aligned}$$

(45)

Thus, if the $S_3$ singlet $\mathbf {1}$ acquires a vev, we can then conclude from Eq. (45) that we are not allowed to have the vev alignments $(1,1),\,(\omega ,1),\,(\omega ^2,1)$ for the two Higgs doublets in the $S_3$ doublet irrep. Similarly, if the $S_3$ singlet $\mathbf {1}^\prime$ acquires a vev, the alignments $(-1,1)$, $(-\omega ,1)$, $(-\omega ^2,1)$ are excluded. Of course, it remains to be seen whether all global minima of the $S_3$-invariant Higgs scalar potential are contained or not in the set of forbidden vev alignments.

4 Conclusions

In this work we have studied the connection between the breaking of flavour symmetries by the Higgs vevs and the existence of residual symmetries in the quark mass matrices. We have performed two tasks. First, in order to avoid nonphysical quark masses and mixing, we have developed a simple but powerful no-go theorem, highlighting the importance of including the transformation properties of the right-handed fields. Second, we have shown that, in many instances, exploring the eigenspaces of all the subgroups of the original flavour group is sufficient to study all the relevant vevs. The vevs of the scalars coupled to the quarks have to break the full flavour symmetry (or break it into a subgroup acting as baryon number) in order to avoid nonphysical quark masses and mixing.

In this context, the notion of full flavour group turns out to be important. While, in general, one refers to the flavour group as the group added to the SM acting globally on the flavour space, the SM gauge group already contains the global hypercharge transformation that should also be taken into account. The inclusion of this additional flavour transformation builds what we call the full flavour group. Then the problem of finding the vevs left invariant by the full flavour group and, therefore, excluded by the theorem, turns into the problem of finding all pseudo-invariant vevs of the flavour group, discussed here for the first time.

As we have shown through some examples, it is possible to find a set of excluded vev alignments (minima or not of the scalar potential) just by determining the eigenvectors of the Higgs representation for each flavour group element. If the global minima are known, as it is the case of 3HDM with $A_4$ or $\Delta (27)$ symmetric potentials, one then needs only to check whether these minima are contained in the set of excluded alignments. For 3HDM with $A_4$ (or $S_4$) and $\Delta (27)$ symmetric potentials, this is indeed verified and thus these models are excluded. In this case, a phenomenologically viable description of the quark sector requires that (1) the symmetry is explicitly broken by new interaction terms, or (2) higher-order interaction terms in the Higgs potential are present, which could then lead to the complete breaking of the $A_4$ or $\Delta (27)$ group upon minimisation of the potentials, or (3) additional non-invariant scalar multiplets are added to the theory. In more realistic but non-minimal models, one or more of these options are necessarily implemented. The same type of analysis can be carried out for other groups and, in particular, for smaller groups which have a more complex scalar potential.

While we only treated quarks in our discussion, the extension of the theorem to the whole fermion sector, including leptons, is straightforward. If neutrinos are Dirac-type particles the analogy is direct. There is the experimental possibility of a massless neutrino, however, as shown in our analysis, all cases with CP violation always imply a massless fermion in each sector. The case where neutrinos are Majorana-type particles is more interesting since there is a larger number of possible implementations. Lepton number is no longer conserved and thus the theorem will be slightly modified.

Note added: While our paper was undergoing the review process, Ref. [31] appeared, in which a classification of lepton mixing matrices is performed based on the assumption that the residual symmetries in the charged-lepton and neutrino mass matrices originate from a finite flavour symmetry group.

Appendices

Appendix A: Linear independence of Yukawa matrices

The proposition presented in Sect. 2.2 relies on the statement that if a set of Higgs doublets $\phi _k$ realises an irreducible representation of the flavour symmetry group $G$, then either all of them decouple from all up-quarks or all down-quarks, or none of them can decouple from them. This statement is stronger than just saying that none of the matrices $\varGamma ^k$ or $\Delta ^k$ can be zero in a given Higgs basis. It says that this cannot happen in any Higgs basis. The basis-invariant formulation of this requirement is that $\varGamma ^k$ or $\Delta ^k$ are either all zeros or are linearly independent.

In this Appendix, we provide an accurate proof of this statement. We start with the following proposition:

Proposition

Let $\{\varGamma ^k\}$, $k = 1,\dots ,N$, be a non-zero element of a complex vector space $\bigotimes _N V_\varGamma$; $V_\varGamma$ is the space of each $\varGamma ^k$. Similarly, let $\{\phi _k\}$, with $k = 1,\dots ,N$, be elements of a complex vector space $\bigotimes _N V_\phi$, where $V_\phi$ is the space of each $\phi _k$. Suppose that a symmetry group $G$ acts in the space of $\{\phi _k\}$, and $\{\phi _k\}$ realise an irrep of $G$. Finally, suppose that there exists an expression ${\mathcal L}$, proportional to $\varGamma ^k \phi _k$ (summation over repeated indices assumed), which is invariant under $G$. Then the $\varGamma ^k$ are linearly independent.

Before proceeding to the proof, a few remarks are in order. First, to make the notation a bit more familiar, we notice that in our case $V_\varGamma = \mathbb {C}^9$ of complex-valued $3\times 3$ matrices $\varGamma ^k$, and $V_\phi = \mathbb {C}^2$, the space of Higgs doublets. However, the proposition itself is not specific to this particular space. Also, we note that the linear independence among $\varGamma ^k$ is a basis-invariant way of saying that no individual $\varGamma ^k$ will ever become zero after any basis change in $\phi _k$. Finally, $\{\varGamma ^k\}$ being a non-zero element of $\bigotimes _N V_\varGamma$ means that not all individual $\varGamma ^k=0$.

Proof

We shall prove the statement by contradiction: assuming that $\varGamma ^k$ are linearly dependent, we will show that none $\{\phi _k\}$ can be an irrep.

Suppose that $\varGamma ^k$ are linearly dependent, namely, that there exist complex coefficients $c_k$, not all of them being zeros, such that $c_1 \varGamma ^1 + \cdots + c_N \varGamma ^N = 0$. Denote the number of linearly independent $\varGamma ^k$ by $p$, with $0 < p < N$. Then it is possible to perform a correlated basis transformation in the spaces of the $\varGamma$ and the $\phi$ leaving invariant $\varGamma ^k \phi _k$, which sets $N-p$ matrices $\varGamma$ to zero:

In this basis, we call the corresponding Higgs doublets $\phi _k$ active for $k = 1,\dots ,p$ and passive for $k = p+1,\dots ,N$. The expression ${\mathcal L}$ is thus written only in terms of active fields. Let us now apply a transformation $g \in G$, which leads to

$$\begin{aligned} \varGamma ^k \phi _k \mapsto \varGamma ^k \phi ^g_k = \varGamma ^k ({\mathcal G}_g)_{kl} \phi _l = \varGamma ^l ({\mathcal G}_g)_{lk} \phi _k. \end{aligned}$$

(47)

The last transformation here is just a relabeling of the indices. Note that we do not require that $\varGamma ^k \phi _k$ itself is invariant under $g$, because ${\mathcal L}$ can contain other overall factors which are also transformed under $g$ (in our particular case, these are left- and right-handed quark fields). These additional factors lead to the appearance of extra transformations (such as ${\mathcal G}_L$ and ${\mathcal G}_R^d$), which are unitary transformations of the space $V_\varGamma$ and transform each individual $\varGamma ^k$, but do not mix different $\varGamma$. In particular, they cannot make a zero $\varGamma ^k$ non-zero, and vice versa, nor can they convert a linearly independent set of $\varGamma$ into a linearly dependent one.

By construction, the superscript $l$ in the last expression of Eq. (47) refers to active fields. Since $\varGamma ^l ({\mathcal G}_g)_{lk}$ is a linear combination of linearly independent $\varGamma$, if $k$ also corresponds to an active field, then this linear combination can stay non-zero, and $({\mathcal G}_g)_{lk}$ can be anything. If $k$ corresponds to a passive field, then this linear combination is zero, which can happen only when $({\mathcal G}_g)_{lk} = 0$ for all active $l$. Since ${\mathcal G}_g$ is unitary, it also follows that $({\mathcal G}_g)_{lk} = 0$ for all passive $l$ and active $k$. In other words, the group transformation ${\mathcal G}_g$ does not mix active and passive fields, and hence it has a block-diagonal form in the space of $\phi$. Repeating this analysis for all $g \in G$, with the same conclusion, we find that active and passive fields represent two invariant subspaces of the space $\{\phi _k\}$. Thus, the representation is reducible. This completes the proof. $\square$

The above proposition can be immediately applied to the one given in Sect. 2.2. Suppose that we pick up Higgs doublets which realise an irrep of the flavour group $G$; this can be either the entire set of doublets of the model or, in the case when the Higgs doublets are in a reducible representation, this can be an irreducible subspace of the Higgs fields. In any of these cases, we apply the above proposition, assuming that $N$ stands for the dimension of the chosen irreducible representation. With this minor modification, we obtain exactly the proposition from the main text.

Appendix B: Block-diagonal CKM matrix

We prove here the assertion made after Eq. (24), that if the generator $\mathcal {G}_{Lg_1}$ given in Eq. (24) is not proportional to the identity matrix, then the CKM matrix is block-diagonal.

The matrix $\mathcal {G}_{Lg_1}$, being a $3\times 3$ unitary matrix, is diagonalizable and we can find three orthonormal eigenvectors. If $\mathcal {G}_{Lg_1}$ is not proportional to the identity, then there is at least one non-degenerate eigenvalue $e^{i\theta }$. Take its associated eigenvector $\mathbf {w}$, for which $\mathcal {G}_{Lg_1}\mathbf {w}=e^{i\theta }\mathbf {w}$. It follows from Eq. (24) that

$$\begin{aligned} \mathcal {G}_{Lg_1}(H_d\,\mathbf {w})=H_d\,\mathcal {G}_{Lg_1}\mathbf {w}=e^{i\theta }(H_d\,\mathbf {w}), \end{aligned}$$

(48)

i.e., $(H_d\,\mathbf {w})\ne \varvec{0}$ is also an eigenvector of $\mathcal {G}_{Lg_1}$ with eigenvalue $e^{i\theta }$. Since this eigenvalue is non-degenerate, $(H_d\,\mathbf {w})$ should be proportional to $\mathbf {w}$: $H_d\,\mathbf {w}=m^2_{d_i}\mathbf {w}$, where obviously $m_{d_i}$ is one of the down-quark masses. Hence, $\mathbf {w}$ fills one of the columns of the matrix $V_d$ which diagonalises $H_d$. Similar arguments make the same vector $\mathbf {w}$ to be in one of the columns of the matrix $V_u$, which diagonalises $H_u$. The CKM matrix $V=V_u^\dag V_d$ then contains one row and one column consisting of zeros, except for a unit entry where they cross. Finally, the vector $\mathbf {w}$ will remain as a common eigenvector of $H_d$ and $H_u$ even if $H_d\mathbf {w}=\varvec{0}$ or $H_u\mathbf {w}=\varvec{0}$ (some massless quark). This proves the assertion.

Appendix C: Yukawa structure

We show here that some combination of representations cannot be assigned to the fields $\bar{Q}_L,\Phi ,d_R$ and $u_R$ in order to construct group invariants in the Yukawa interactions of Eq. (1). For instance, it is not possible that only one of $\bar{Q}_L,\Phi ,d_R$ or $u_R$ is assigned to a faithful irrep or contains a faithful irrep.

We begin by reviewing the following result: given two vectors (fields) $x$ and $y$ that transform under irreducible representations $\mu$ and $\nu$ of $G$, the unique invariant present in $x \times y$ is $x \cdot y=x_i\,y_i$, only possible when $\mu =\nu ^*$. The proof relies on the Schur lemmas. The invariant in $x\times y$ can be written as $I=x_i\,y_j\,C_{ij}$ where $C_{ij}$ are the Clebsch–Gordan coefficients that connect $x\times y$ to the trivial representation. Invariance of $I$ requires

$$\begin{aligned} (D^{(\mu )}(g))^TC D^{(\nu )}(g)=C,\quad \text{ for } \text{ all }~ g \,\mathrm{in}\, G. \end{aligned}$$

(49)

We can rewrite it as $C D^{(\nu )}(g)=(D^{(\mu )}(g))^*C$. One of the Schur lemmas requires that $C\ne \varvec{0}$ only if $\mu \simeq \nu ^*$. However, if $\mu =\nu ^*$, another Schur lemma ensures that $C={1}\mathrm{l}$. Now, this result also applies if $x$ and $y$ are reducible and composed of more than one irreducible piece: complex conjugate irreps need to be contracted to build invariants.

Let us now analyse $\bar{Q}_L\times \Phi \times d_R$. Suppose that $Q_L$ and $\Phi$ are assigned to unfaithful representations and $d_R$ to a faithful irrep. Then the representation of $\bar{Q}_L\times \Phi$ is also unfaithful and does not contain any faithful representation that can be contracted to $d_R$.