Orbit closures and invariants

Abstract

Let G be a reductive linear algebraic group, H a reductive subgroup of G and X an affine G-variety. Let $X^H$ denote the set of fixed points of H in X, and $N_G(H)$ the normalizer of H in G. In this paper we study the natural map of quotient varieties $\psi _{X,H}:X^H/N_G(H) \rightarrow X/G$ induced by the inclusion $X^H \subseteq X$. We show that, given G and H, $\psi _{X,H}$ is a finite morphism for all affine G-varieties X if and only if H is a G-completely reducible subgroup of G (in the sense defined by Serre); this was proved in characteristic 0 by Luna in the 1970s. We discuss some applications and give a criterion for $\psi _{X,H}$ to be an isomorphism. We show how to extend some other results in Luna’s paper to positive characteristic and also prove the following theorem. Let H and K be reductive subgroups of G; then the double coset HgK is closed for generic $g\in G$ if and only if $H\cap gKg^{-1}$ is reductive for generic $g\in G$.

1 Introduction

The purpose of this paper is to establish some results in geometric invariant theory over fields of positive characteristic, where tools from characteristic 0—such as Luna’s Étale Slice Theorem—are not available. In particular, we prove the following theorem and give some applications (see Sect. 2 for precise definitions of terms). Let k be an algebraically closed field of characteristic $p\ge 0$.

Theorem 1.1

Suppose G is a reductive linear algebraic group over k and H is a reductive subgroup of G. Then the following are equivalent:

• (i) H is G-completely reducible;

• (ii) $N_G(H)$ is reductive and, for every affine G-variety X, the natural map of quotients $\psi _{X,H}:X^H/N_G(H) \rightarrow X/G$ is a finite morphism (here $X^H$ denotes the H-fixed points in X).

The study of closed orbits is central in geometric invariant theory—the closed orbits for G in X parametrise the points of the quotient variety X / G. An important piece of the proof of Theorem 1.1 is Proposition 4.1, which gives a connection between the closed G-orbits in X and the closed H-orbits in X; cf. [2, 33, 48] and [7], for example.

Theorem 1.1 reduces to the main result in Luna’s paper [33] when k has characteristic 0, because condition (i) and the first hypothesis of (ii) are automatic in characteristic 0 if H is already assumed to be reductive. Luna’s methods use the powerful machinery of étale slices, based on his celebrated “Étale Slice Theorem” [32]; see Sect. 3.1 below for more on étale slices. Many useful consequences flow from the existence of an étale slice (see Proposition 3.1 below, for example). Although étale slices sometimes exist in positive characteristic [1], in general they do not. Our methods differ from Luna’s in that they apply equally well in all characteristics. These methods also allow us to provide extensions to positive characteristic of other results from [33] (see Proposition 3.10, Remark 4.2(i) and Proposition 4.7).

The work in this paper fits into a broad continuing programme of taking results about algebraic groups and related structures from characteristic zero and proving analogues in positive characteristic. A basic problem with this process is that results—such as the existence of an étale slice—that are true when $p=0$ may simply fail when $p>0$ (cf. Examples 3.2, 8.1, 8.2 and 8.3); a further illustration of this in the context of this paper is that a reductive group may fail to be linearly reductive (recall that a linear algebraic group is called reductive if it has trivial unipotent radical, and linearly reductive if all its rational representations are semisimple). When $p=0$, a connected group is linearly reductive if and only if it is reductive, whereas if $p>0$ a connected group is linearly reductive if and only if it is a torus [42]. Even if a result remains true in positive characteristic, it may be much harder to prove, an example here being the problem of showing that the ring of invariants $R^G$ is finitely generated, where R is a finitely-generated k-algebra and $G \subseteq {{\,\mathrm{Aut}\,}}(R)$ is reductive. This was resolved in characteristic 0 in the 1950s, but not in positive characteristic until the 1970s (see the introduction to Haboush’s paper [22]).

In some contexts in positive characteristic where the hypothesis of reductivity is too weak and linear reductivity is too strong, it has been found that a third notion, that of G-complete reducibility, provides a good balance (cf. [37, Cor. 1.5]) and our main theorem is another example of this phenomenon. See Sect. 2.4 for the definition. The idea is that when $p=0$ there is no distinction between demanding that a subgroup H of a reductive group G is reductive or linearly reductive or G-completely reducible, but there is a huge difference in positive characteristic. The notion of complete reducibility was introduced by Serre [52] and Richardson¹ [49], and over the past twenty years or so has found many applications in the theory of algebraic groups, their subgroup structure and representation theory, geometric invariant theory, and the theory of buildings: for examples, see [2, 5, 9, 30, 35, 36, 39, 56,57,58].

The paper is set out as follows. Section 2 contains preparatory material from geometric invariant theory and the theory of complete reducibility. The proof of Theorem 1.1 contains three main ingredients, each dealt with in a separate section. In Sect. 3 we build on work of Bardsley and Richardson to establish the important technical result Proposition 3.10, which gives a criterion for a map of quotient varieties to be finite. In Sect. 4 we carry out our analysis of the closed G- and H-orbits and show that $\psi _{X,H}$ is quasi-finite if H is G-completely reducible (Theorem 4.4). In Sect. 5 we show that the image of $\psi _{X,H}$ is closed (Theorem 5.1). The key idea here is to consider the map of projectivisations ${{\mathbb {P}}}(X^H)\rightarrow {{\mathbb {P}}}(X)$ induced by the inclusion of $X^H$ in X when X is a G-module; the G-complete reducibility of H guarantees that we get a well-defined map of quotient varieties ${{\mathbb {P}}}(X^H)/G\rightarrow {{\mathbb {P}}}(X)/G$. Section 6 draws these strands together and completes the proof of Theorem 1.1 using Proposition 3.8 (a variation on Zariski’s Main Theorem).

Section 7 gives a criterion for $\psi _{X,H}$ to be an isomorphism onto its image (Theorem 7.2). In Sect. 8 we use representation theory to construct some examples relevant to Theorem 1.1. In Sect. 9 we give a criterion (Theorem 9.1) for generic double cosets HgK of G to be closed, where H and K are reductive subgroups of G. Luna proved a stronger result [31] in characteristic 0 using étale slice methods, but our techniques work when étale slices are not available. We give some applications of Theorem 9.1 (Examples 9.11 and 9.12); these serve as applications of Theorem 1.1 as well. We finish in Sect. 10 by using the theory we have developed to prove some results on complete reducibility.

2 Notation and preliminaries

2.1 Notation

Our basic references for the theory of linear algebraic groups are the books [10] and [54]. Unless otherwise stated, we work over a fixed algebraically closed field k with no restriction on the characteristic. By a variety we mean a quasi-projective variety over k, and we identify a variety X with its set of k-points. For a linear algebraic group G over k, we let $G^0$ denote the connected component of G containing the identity element 1 and $R_u(G) \unlhd G^0$ denote the unipotent radical of G. We say that G is reductive if $R_u(G) = \{1\}$; note that we do not require a reductive group to be connected. When we discuss subgroups of G, we really mean closed subgroups; for two such subgroups H and K of G, we set $HK:=\{hk\mid h \in H, k\in K\}$. We denote the centralizer of a subgroup H of G by $C_G(H)$, and the normalizer by $N_G(H)$. All group actions are left actions unless otherwise indicated.

We make repeated use of the following result [35, Lemma 6.8]: if G is reductive and if H is a reductive subgroup of G then $N_G(H)^0= H^0C_G(H)^0$.

Given a linear algebraic group G, let Y(G) denote the set of cocharacters of G, where a cocharacter is a homomorphism of algebraic groups $\lambda :k^* \rightarrow G$. Note that since the image of a cocharacter is connected, we have $Y(G) = Y(G^0)$. A linear algebraic group G acts on its set of cocharacters: for $g \in G$, $\lambda \in Y(G)$ and $a \in k^*$, we set $(g\cdot \lambda )(a) = g\lambda (a)g^{-1}$.

Given an affine variety X over k, we denote the coordinate ring of X by k[X] and the function field of X (when X is irreducible) by k(X). Given $x \in X$, we let $T_x(X)$ denote the tangent space to X at x. Recall that for a linear algebraic group G, $T_1(G)$ has the structure of a Lie algebra, which we also denote by ${{\,\mathrm{Lie}\,}}(G)$ or ${{\mathfrak {g}}}$. Given a morphism $\phi :X \rightarrow Y$ of affine varieties X and Y and a point $x \in X$, we let $d_x\phi :T_x(X) \rightarrow T_{\phi (x)}(Y)$ denote the differential of $\phi$ at x. We say that X is a G-variety if the linear algebraic group G acts morphically on X. If X is affine then the action of G on X gives a linear action of G on k[X], defined by $(g\cdot f)(x) = f(g^{-1}\cdot x)$ for all $g \in G$, $f \in k[X]$ and $x \in X$. Given a G-variety X and $x \in X$, we denote the G-orbit through x by $G\cdot x$ and the stabilizer of x in G by $G_x$. If $x,y \in X$ are two points on the same G-orbit, then we sometimes say x and y are G-conjugate. For $x\in X$, we denote the orbit map $G\rightarrow G\cdot x$, $g\mapsto g\cdot x$ by $\kappa _x$; we say the orbit $G\cdot x$ is separable if $\kappa _x$ is separable. We denote by $X^G$ the set of G-fixed points in X, and by $k[X]^G$ the ring of G-invariant functions in k[X].

Given a morphism of varieties $f:V\rightarrow W$, define e(v) for $v\in V$ to be $\mathrm{max}(\mathrm{dim}(Z))$, where Z ranges over the irreducible components of $f^{-1}(f(v))$ that contain v. By [10, AG.10.3], e(v) is an upper semi-continuous function of v. This implies the following useful result about dimensions of stabilizers for a G-variety X [44, Lemma 3.7(c)]: for any $r\in {{\mathbb {N}}}\cup \{0\}$, the set $\{x\in X\mid \mathrm{dim}(G_x)\ge r\}$ is closed. We deduce the lower semi-continuity of orbit dimension: that is, for any $r\in {{\mathbb {N}}}\cup \{0\}$, the set $\{x\in X\mid \mathrm{dim}(G\cdot x)\le r\}$ is closed. In particular, the set $\{x\in X\mid \mathrm{dim}(G\cdot x) \text{ is } \text{ maximal }\}$ is open. We also need an infinitesimal version of these results. Given a variety Z, we denote the (reduced) tangent bundle of Z by TZ; we may identify TZ with the set of pairs $\{(z,v)\mid z\in Z, v\in T_z(Z)\}$, and we have a canonical embedding from Z to TZ given by $z\mapsto (z,0)$. (The tangent bundle is constructed in [10, AG.16] as a possibly non-reduced scheme over k; here we take the tangent bundle to be the corresponding reduced scheme.) If $\psi :Z\rightarrow W$ is a morphism of varieties then we have a map $d\psi :TZ\rightarrow TW$ given by $d\psi (z,v)= (\psi (z), d_z\psi (v))$.

Lemma 2.1

For any $r\in {{\mathbb {N}}}\cup \{0\}$, the set $\{x\in X\mid \mathrm{dim}(G_x)+ \mathrm{dim}(\mathrm{ker}(d_1\kappa _x))\ge r\}$ is closed.

Proof

Define $\alpha :G\times X\rightarrow X\times X$ by $\alpha (g,x)= (g\cdot x, x)$. We obtain a morphism $d\alpha$ from $T(G\times X)\cong TG\times TX$ to $T(X\times X)\cong TX\times TX$. Let $x\in X$ and consider the point $y:= ((x,0), (x,0))\in TX\times TX$. Now $(d\alpha )^{-1}(y)$ is a closed subset $C_y$ of $TG\times TX$; it is clear that $C_y= \{((g,v),(x,0))\mid g\in G_x, v\in \mathrm{ker}(d_1\kappa _x))\}$. Each irreducible component of this set has dimension $e'(y):= \mathrm{dim}(G_x)+ \mathrm{dim}(\mathrm{ker}(d_1\kappa _x))$.

Define a function $s:X\rightarrow TG\times TX$ by $s(x)= ((1,0), (x,0))$. We identify X with a closed subset of $TX\times TX$ via the embedding $x\mapsto ((x,0), (x,0))$. Since s is a morphism, we deduce from the upper semi-continuity of the function e(v)—taking $(V,W,f)= (TG\times TX, TX\times TX, d\alpha )$—that the function $e'(y)$ is also upper semi-continuous. The result now follows. $\square$

A morphism $\phi :X \rightarrow Y$ of affine varieties is said to be finite if the coordinate ring k[X] is integral over the image of the comorphism $\phi ^*:k[Y] \rightarrow k[X]$. Finite morphisms are closed [40, Proposition I.7.3(i)]; in particular, a dominant finite morphism is surjective. A morphism of affine varieties is called quasi-finite if its fibres are finite; finite morphisms are always quasi-finite [40, Proposition I.7.3(ii)], but the converse is not true. A dominant morphism $\phi :X\rightarrow Y$ of irreducible varieties is called birational if the comorphism induces an isomorphism of function fields $k(X)\cong k(Y)$. Given an irreducible affine variety X, we can form the normalization of X by considering the normal affine variety ${\widetilde{X}}$ whose coordinate ring is the integral closure of k[X] in the function field k(X). The normalization map $\nu _X:{\widetilde{X}} \rightarrow X$ is, by construction, finite, birational and surjective.

Remark 2.2

We record an observation which we use several times in the sequel. Let $\phi :X\rightarrow Y$ and $\psi :Y\rightarrow Z$ be morphisms of affine varieties with $\psi \circ \phi$ finite. Then it is easy to see that:

• (i) $\phi$ is finite;

• (ii) if $\phi$ is dominant then $\psi$ is finite.

We say that a property P(x) holds for generic$x\in X$ if there is an open dense subset U of X such that P(x) holds for all $x\in U$.

For the remainder of the paper, we fix the convention that G denotes a reductive linear algebraic group over k.

2.2 Group actions and quotients

The main result of this paper, Theorem 1.1, concerns quotients of affine varieties by reductive algebraic group actions. Let X be an affine G-variety. As noted above, G acts on k[X], and we can form the subring $k[X]^G \subseteq k[X]$ of G-invariant functions on X. It follows from [43] and [22] that $k[X]^G$ is finitely generated, and hence we can form an affine variety denoted X / G with coordinate ring $k[X/G] = k[X]^G$. Moreover, the inclusion $k[X]^G \hookrightarrow k[X]$ gives rise to a morphism from X to X / G, which we shall denote by $\pi _{X,G}:X \rightarrow X/G$. The map $\pi _{X,G}$ has the following properties [41, Theorem  A.1.1], [44, Theorem 3.5], [1, Sect. 2]:

• (i) $\pi _{X,G}$ is surjective;

• (ii) $\pi _{X,G}$ is constant on G-orbits in X;

• (iii) $\pi _{X,G}$ separates disjoint closed G-invariant subsets of X;

• (iv) each fibre of $\pi _{X,G}$ contains a unique closed G-orbit, and $\pi _{X,G}$ determines a bijective map from the set of closed G-orbits in X to X / G;

• (v) X / G is a categorical quotient of X: that is, for every variety V and every morphism $\psi :X\rightarrow V$ which is constant on G-orbits, there is a unique morphism $\psi _G:X/G\rightarrow V$ such that $\psi = \psi _G\circ \pi _{X,G}$.

(This means $\pi _{X,G}$ is a good quotient in the sense of [44, Chapter 3, Sect. 4, p 57]. More generally, if X is a quasi-projective G-variety and $\pi$ is a map from X to another quasi-projective variety Y then we call $\pi$ a good quotient if it is an affine map and satisfies (i)–(v) above.) We say that $\pi _{X,G}:X\rightarrow X/G$ is a geometric quotient if the fibres of $\pi _{X,G}$ are precisely the G-orbits. This is the case if and only if every G-orbit is closed (for instance, if every G-orbit has the same dimension—e.g., if G is finite).

If $\phi :Y\rightarrow X$ is a G-equivariant morphism of affine G-varieties, then the restriction of the comorphism to $k[X]^G$ induces a natural morphism from Y / G to X / G, which we shall denote by $\phi _G$. In a special case of this construction, we have the following result, which follows from [44, Theorem 3.5, Lemma 3.4.1].

Lemma 2.3

Let X be an affine G-variety and let $i:Y \rightarrow X$ be an embedding of a closed G-stable subvariety Y in X. Then $\pi _{X,G}(Y)$ is closed in X / G. Moreover, the induced map $i_G:Y/G\rightarrow X/G$ is injective and finite.

Remark 2.4

If $\mathrm{char}(k)= 0$ then $i_G$ is an isomorphism onto its image. This need not be the case in positive characteristic: see Example 3.2.

We record some other useful results. First, note that if G is a finite group, then the map $\pi _G$ above is a finite morphism. To see this, let $f\in k[X]$ and let T be an indeterminate. Then the polynomial $F(T):=\prod _{g\in G} (T-g\cdot f)\in (k[X])[T]$ is monic and has coefficients in $k[X]^G$, and $F(f) = 0$. This shows that k[X] is integral over $k[X]^G$, which gives the claim.

If X is irreducible and normal then X / G is normal [1, 2.19(a)], while if G is connected then $k[X]^G$ is integrally closed in k[X] [1, 2.4.1].

Now suppose H is a subgroup of G such that the normalizer $N_G(H)$ is reductive. Then the inclusion $X^H \subseteq X$ induces a map of quotients $\psi _{X,H}:X^H/N_G(H) \rightarrow X/G$. Theorem 1.1 asserts that when H is a G-completely reducible subgroup of G (in the sense of Sect. 2.4 below), this map is always a finite morphism.

For technical reasons, we sometimes need to work with affine G-varieties satisfying an extra property.

Definition 2.5

Let X be an affine G-variety. We denote by $X_{\mathrm{cl}}$ the closure of the set $\{x\in X\mid G\cdot x\ \text{ is } \text{ closed }\}$. Following Luna [32, Sec. 4], we say that X has good dimension (“bonne dimension”) if $X_{\mathrm{cl}}= X$. We say that x is a stable point of X for the G-action if $\mathrm{dim}(G\cdot x)$ is maximal and $G\cdot x$ is closed [44, Ch. 3, Sect. 4], [41, Ch. 1, Sect. 4].

Remark 2.6

The set of stable points is open [44, Ch. 3, Sect. 4], [41, Ch. 1, Sect. 4] (this is true even without the assumption that G is reductive). Since the set $\{x\in X\mid \mathrm{dim}(G\cdot x) \text{ is } \text{ maximal }\}$ is open, it follows that if X is irreducible then X has good dimension if and only if there exists a stable point. Moreover, if X has good dimension then generic fibres of $\pi _{X,G}:X\rightarrow G$ are orbits of G. Hence if X is irreducible then $\mathrm{dim}(X/G)= \mathrm{dim}(X)- m$, where m is the maximal orbit dimension.

Lemma 2.7

Let X be an irreducible affine G-variety with good dimension. Then $k(X/G)= k(X)^G$. Moreover, $\pi _{X,G}$ is separable.

Proof

It is clear that k(X / G) is a subfield of $k(X)^G$. Conversely, let $f\in k(X)^G$. Set

$$\begin{aligned} U= \{x\in X\mid \hbox {there exists }h_1,h_2\in k[X]\hbox { such that }f= h_1/h_2\hbox { and }h_2(x)\ne 0\}. \end{aligned}$$

Then U is a nonempty open subset of X, and clearly U is G-stable. Hence $C:= X\backslash U$ is closed and G-stable. As X has good dimension, there exists $0\ne h\in k[X]^G$ such that $h|_C= 0$. Now f is a globally defined regular function on the corresponding principal open set $X_h$, so $f\in k[X_h]= k[X][1/h]$. Hence $\displaystyle f= \frac{f'}{h^r}$ for some $f'\in k[X]$ and some $r\ge 0$. Then $f'$ is G-invariant, since f is, so $f\in k(X/G)$.

The second assertion is [1, 2.1.9(b)]. Note that separability can fail if X does not have good dimension: see [38]. $\square$

Lemma 2.8

Let $\phi :X\rightarrow Y$ be a finite surjective G-equivariant map of affine G-varieties.

• (i) For all $x\in X$, $G\cdot x$ is closed if and only if $G\cdot \phi (x)$ is closed. Moreover, if $y\in Y$ and $G\cdot y$ is closed then $\phi ^{-1}(G\cdot y)$ is a finite union of G-orbits, each of which is closed and has the same dimension as $G\cdot y$.

• (ii) The map $\phi _G:X/G\rightarrow Y/G$ is quasi-finite.

• (iii) X has good dimension if and only if Y does.

Proof

If $x\in X$ and $G\cdot x$ is closed then $G\cdot \phi (x)= \phi (G\cdot x)$ is closed, as $\phi$ is finite. Conversely, let $y\in Y$ such that $G\cdot y$ is closed, and let $n= \mathrm{dim}(G\cdot y)$. Let $x\in \phi ^{-1}(G\cdot y)$. Then $\mathrm{dim}(G_x)\le \mathrm{dim}(G_y)$, so $\mathrm{dim}(G\cdot x)\ge \mathrm{dim}(G\cdot y)= n$. But $\phi$ is finite, so every irreducible component of $\phi ^{-1}(G\cdot y)$ has dimension n. It follows that $\mathrm{dim}(G\cdot x)= n$ and $G\cdot x$ is a union of irreducible components of $\phi ^{-1}(G\cdot y)$; in particular, $G\cdot x$ is closed. This proves (i). Part (iii) now follows.

To prove part (ii), let $x\in X$, $y\in Y$ such that $\phi _G(\pi _{X,G}(x))= \pi _{Y,G}(y)$. Without loss of generality, we can assume that $G\cdot x$ and $G\cdot y$ are closed. Now $G\cdot \phi (x)$ is closed by (i), so we must have $G\cdot \phi (x)= G\cdot y$, so $x\in \phi ^{-1}(G\cdot y)$. But $\phi ^{-1}(G\cdot y)$ is a finite union of G-orbits by (i), so we are done. $\square$

Lemma 2.9

Let $\phi :X\rightarrow Y$ be a finite birational G-equivariant morphism of irreducible affine G-varieties. If one of X or Y has good dimension then $\phi _G:X/G\rightarrow Y/G$ is birational.

Proof

By Lemma 2.8(iii), if one of X or Y has good dimension then they both do. It follows from Lemma 2.7 that $k(Y/G)= k(X/G)= k(X)^G$; hence $\phi _G$ is birational. $\square$

Later we also need some material on constructing quotients of projective varieties by actions of reductive groups, but we delay this until Sect. 5.

Suppose H is a subgroup of G. Recall that the quotient G / H (which as a set is just the coset space) has the structure of a quasi-projective homogeneous G-variety, and H is the stabilizer of the image of $1 \in G$ under the natural map $\pi _{G,H}:G \rightarrow G/H$. Richardson has proved the following in this situation ([47, Theorem  A]; see also [23]).

Theorem 2.10

Suppose H is a subgroup of G. Then G / H is an affine variety if and only if H is reductive.

Recall also that the Zariski topology on G / H is the quotient topology: that is, a subset $S \subseteq G/H$ is closed in G / H if and only if $\pi _{G,H}^{-1}(S)$ is closed in G. We need a technical result.

Lemma 2.11

Let H be a reductive subgroup of G. There exist a G-module Y and a nonempty open subset U of $Y^H$ such that the following hold:

• (i) $G_y= H$ for all $y\in U$;

• (ii) $G\cdot y$ is closed for all $y\in U$;

• (iii) $N_G(H)\cdot y$ is closed for all $y\in U$.

Proof

Since H is reductive, G / H is affine. The group G acts on G / H by left multiplication. Let $x_0= \pi _{G,H}(1)$; then $G_{x_0}= H$. If K is a reductive subgroup of G containing H then $K\cdot \pi _{G,H}(1)= \pi _{G,H}(K)$ is closed, as K is a closed subset of G that is stable under right multiplication by H. We can embed G / H equivariantly in a G-module X. By the lower semi-continuity of orbit dimension, there is a nonempty open subset $U_1$ of $X^H$ such that $\mathrm{dim}(G_x)= \mathrm{dim}(H)$ for all $x\in U_1$—so $G_x$ is a finite extension of H for all $x\in U_1$. If $\mathrm{char}(k)= 0$ then we can conclude from Proposition 3.1 that there is an open neighbourhood O of $x_0$ such that $G_x\le H$ for all $x\in O$. It then follows (applying the arguments for (ii) and (iii) below) that we can take Y to be X and U a suitable nonempty open subset of $X^H\cap O$. In general, however, we need a slightly more complicated construction.

Let Y be the G-module $X\oplus X$. Note that $Y^H= X^H\oplus X^H$ and for any $(y_1,y_2)\in Y^H$, $G_{(y_1,y_2)}= G_{y_1}\cap G_{y_2}$. We show that Y has the desired properties. For each $r\ge 0$, define

$$\begin{aligned} C_r= \{y\in U_1\times U_1\mid |G_y:H|\ge r\}. \end{aligned}$$

Then $C_r$ is empty for all but finitely many r by [37, Lemma 2.2 and Defn. 2.3]. Moreover, $C_r$ is constructible. For let

$$\begin{aligned} {\widetilde{C}}_r= \{(y,g_1,\ldots , g_r)\mid y\in U_1\times U_1, g_1,\ldots , g_r\in G_y, g_jg_i^{-1}\not \in H\ \hbox { for}\ 1\le i,j\le r\}; \end{aligned}$$

then $C_r$ is the image of ${\widetilde{C}}_r$ under projection onto the first factor. Set $D_r= C_r\backslash C_{r+1}$. Then the nonempty $D_r$ form a finite collection of disjoint constructible sets that cover the irreducible set $U_1\times U_1$, so $D_s$ contains a nonempty open subset $U_2$ of $U_1\times U_1$ for precisely one value of s.

We show that $s= 1$. Suppose not. Choose $y= (x_1,x_2)\in U_2$. Let $g_1, g_2, \ldots , g_r$ be coset representatives for $G_{x_1}/H$ with $g_1\in H$. Note that $U_3= \{x\in U_1\mid (x_1,x)\in U_2\}$ is an open dense subset of $X^H$. Let $z= (x_1,x)\in U_2$. Then our hypothesis means that $g\cdot (x_1,x)= (x_1,x)$ for some $g\not \in H$. Now g must fix $x_1$, so $g\in g_iH$ for some $i\ge 1$; in fact, $i\ge 2$ since $g\not \in H$. It follows that $g_i$ fixes $(x_1,x)$ since H fixes $(x_1,x)$, so $g_i$ fixes x. But $\bigcup _{j=2}^r (X^{g_j}\cap X^H)$ is a proper closed subset of $X^H$ as none of the $g_j$ for $j\ge 2$ fixes $x_0$, so we have a contradiction. We conclude that $s= 1$ after all. Hence $G_y= H$ for all $y\in U_2$.

Set $y_0= (x_0,0)$. The orbit $N_G(H)\cdot x_0$ is closed in G / H, so the orbit $N_G(H)\cdot y_0$ is closed in $Y^H$. Moreover, $N_G(H)_{y_0}= H$, so $N_G(H)\cdot y_0$ has maximal dimension among the $N_G(H)$-orbits on $Y^H$. Hence $y_0$ is a stable point of $Y^H$ for the $N_G(H)$-action. A similar argument shows that $y_0$ is a stable point of $\overline{G\cdot Y^H}$ for the G-action. Since the set of stable points is open in each case, we can find a nonempty open subset U of $U_2$ such that (ii) and (iii) hold for U; then (i) holds for U by construction. This completes the proof. $\square$

2.3 Cocharacters, G-actions and R-parabolic subgroups

Suppose that X is a G-variety. For any cocharacter $\lambda \in Y(G)$ and $x \in X$ we can define a morphism $\psi = \psi _{x,\lambda }:k^* \rightarrow X$ by $\psi (a) = \lambda (a)\cdot x$ for each $a \in k^*$. We say that the limit $\lim _{a\rightarrow 0} \lambda (a)\cdot x$exists if $\psi$ extends to a morphism ${\overline{\psi }}: k \rightarrow X$. If the limit exists, then the extension ${\overline{\psi }}$ is unique, and we set $\lim _{a\rightarrow 0} \lambda (a)\cdot x = {\overline{\psi }}(0)$. It is clear that, for any G and X, if there exists $\lambda \in Y(G)$ such that $\lim _{a\rightarrow 0}\lambda (a)\cdot x$ exists but lies outside $G\cdot x$, then $G\cdot x$ is not closed in X.

A subgroup P of G is called a parabolic subgroup if the quotient G / P is complete; this is the case if and only if G / P is projective. If G is connected and reductive, then all parabolic subgroups of G have a Levi decomposition$P = R_u(P)\rtimes L$, where the reductive subgroup L is called a Levi subgroup of P. In this case, the unipotent radical $R_u(P)$ acts simply transitively on the set of Levi subgroups of P, and given a maximal torus T of P there exists a unique Levi subgroup of P containing T. For these standard results see [10, 11] or [54] for example. It is possible to extend these ideas to a non-connected reductive group using the formalism of R-parabolic subgroups described in [5, Sec. 6]. We give a brief summary; see loc. cit. for further details. Given a cocharacter $\lambda \in Y(G)$, we have:

• (i) $P_\lambda := \{g \in G \mid \lim _{a\rightarrow 0}\lambda (a)g\lambda (a)^{-1}\text { exists}\}$ is a parabolic subgroup of G; we call a parabolic subgroup arising in this way an R-parabolic subgroup of G.

• (ii) $L_\lambda := C_G(\lambda ) = \{g \in G \mid \lim _{a\rightarrow 0}\lambda (a)g\lambda (a)^{-1}= g\}$ is a Levi subgroup of $P_\lambda$; we call a Levi subgroup arising in this way an R-Levi subgroup of G.

• (iii) $R_u(P_\lambda ) = \{g \in G \mid \lim _{a\rightarrow 0}\lambda (a)g\lambda (a)^{-1}= 1\}$.

The R-parabolic (resp. R-Levi) subgroups of a connected reductive group G are the same as the parabolic and Levi subgroups of G. Moreover, the results listed above for parabolic and Levi subgroups of connected reductive algebraic groups also hold for R-parabolic and R-Levi subgroups of non-connected reductive groups; that is, the unipotent radical $R_u(P)$ acts simply transitively on the set of R-Levi subgroups of an R-parabolic subgroup P, and given a maximal torus T of P there exists a unique R-Levi subgroup of P containing T.

Now, if H is a reductive subgroup of G and $\lambda \in Y(H)$, then $\lambda$ gives rise in a natural way to R-parabolic and R-Levi subgroups of both G and H. In such a situation, we reserve the notation $P_\lambda$ (resp. $L_\lambda$) for R-parabolic (resp. R-Levi) subgroups of G, and use the notation $P_\lambda (H)$, $L_\lambda (H)$, etc. to denote the corresponding subgroups of H. Note that for $\lambda \in Y(H)$, it is obvious from the definitions that $P_\lambda (H) = P_\lambda \cap H$, $L_\lambda (H) = L_\lambda \cap H$ and $R_u(P_\lambda (H)) = R_u(P_\lambda ) \cap H$.

2.4 G-complete reducibility

Our main result, and many of the intermediate ones, uses the framework of G-complete reducibility introduced by Serre [52], which has been shown to have geometric implications in [5] and subsequent papers. We give a short recap of some of the key ideas concerning complete reducibility.

Let H be a subgroup of G. Following Serre (see, for example, [52]), we say that H is G-completely reducible (G-cr for short) if whenever $H \subseteq P$ for an R-parabolic subgroup P of G, there exists an R-Levi subgroup L of P such that $H \subseteq L$. For example, if $G= {{\,\mathrm{SL}\,}}_n(k)$ or ${{\,\mathrm{GL}\,}}_n(k)$ then H is G-cr if and only if the inclusion of H is completely reducible in the usual sense of representation theory. If H is G-cr then H is reductive, while if H is linearly reductive then H is G-cr (see [5, Sects.  2.4, 6]). Hence in characteristic 0, H is G-cr if and only if H is reductive.

In [2] and [37] it was shown that the notion of complete reducibility is useful when one considers G-varieties and, as explained in the introduction, one of the purposes of this paper is to expand upon this theme.

The geometric approach to complete reducibility outlined in [5] rests on the following construction, which was first given in this form in [9]. Given a subgroup H of a reductive group G and a positive integer n, we call a tuple of elements ${\mathbf {h}} \in H^n$ a generic tuple for H if there exists a closed embedding of G in some ${{\,\mathrm{GL}\,}}_m(k)$ such that ${\mathbf {h}}$ generates the associative subalgebra of $m\times m$ matrices spanned by H [9, Defn. 5.4]. A generic tuple for H always exists for sufficiently large n. Suppose ${\mathbf {h}} \in H^n$ is a generic tuple for H; then in [9, Theorem  5.8(iii)] it is shown that H is G-completely reducible if and only if the G-orbit of ${\mathbf {h}}$ in $G^n$ is closed, where G acts on $G^n$ by simultaneous conjugation.

2.5 Optimal cocharacters

Let X be an affine G-variety. The classic Hilbert–Mumford Theorem [28, Theorem  1.4] says that via the process of taking limits, the cocharacters of G can be used to detect whether or not the G-orbit of a point in X is closed. Kempf strengthened the Hilbert–Mumford Theorem in [28] (see also [24, 41, 51]), by developing a theory of “optimal cocharacters” for non-closed G-orbits. We give an amalgam of some results from Kempf’s paper; see [28, Theorem  3.4, Cor. 3.5] (and see also [9, Sect. 4] for the case of non-connected G).

Theorem 2.12

Let $x \in X$ be such that $G\cdot x$ is not closed, and let S be a closed G-stable subset of X which meets $\overline{G\cdot x}$. Then there exists an R-parabolic subgroup P(x) of G and a nonempty subset $\Omega (x) \subseteq Y(G)$ such that:

• (i) for all $\lambda \in \Omega (x)$, $\lim _{a\rightarrow 0} \lambda (a)\cdot x$ exists, lies in S, and is not G-conjugate to x;

• (ii) for all $\lambda \in \Omega (x)$, $P_\lambda = P(x)$;

• (iii) $R_u(P(x))$ acts simply transitively on $\Omega (x)$;

• (iv) $G_x \subseteq P(x)$.

3 Preparatory results

In this section we collect some results concerning algebraic group actions on varieties which will be useful in the rest of the paper. Recall our standing assumption that G is a reductive group.

3.1 Étale slices

Étale slices are a powerful tool in geometric invariant theory. Let X be an affine G-variety and let $x\in X$ such that $G\cdot x$ is closed. Luna introduced the notion of an étale slice through x [32, III.1]: this is a locally closed affine subvariety S of X with $x\in S$ satisfying certain properties. He proved that an étale slice through x always exists when the ground field has characteristic 0. Bardsley and Richardson later defined étale slices in arbitrary characteristic [1, Defn. 7.1] and gave some sufficient conditions for an étale slice to exist [1, Propns. 7.3–7.6]. If an étale slice exists through x, the orbit $G\cdot x$ must be separable. We record an important consequence of the étale slice theory [1, Proposition 8.6].

Proposition 3.1

Let X be an affine G-variety and let $x\in X$ such that $G\cdot x$ is closed and there is an étale slice through x. Then there is an open neighbourhood U of x such that for all $u\in U$, $G_u$ is conjugate to a subgroup of $G_x$.

The following example, based on a construction from [37, Example 8.3], shows that this result need not hold when there is no étale slice.

Example 3.2

Let $G= {{\,\mathrm{SL}\,}}_2(k)$ and let $H= C_p\times C_p= \langle \gamma _1,\gamma _2\mid \gamma _1^p= \gamma _2^p= [\gamma _1,\gamma _2]= 1\rangle$. Define $f:k\times H\rightarrow k\times G$ by $f(x,h)= (x,f_x(h))$, where $f_x(\gamma _1^{h_1}\gamma _2^{h_2}):= \left( \begin{array}{cc} 1 &{} \quad h_1x+ h_2x^2 \\ 0 &{}\quad 1 \end{array} \right)$. Set $K_x= \mathrm{im}(f_x)$. Note that for each $x\in k$, there are only finitely many $x'\in k$ such that $K_x$ and $K_{x'}$ are G-conjugate. Define actions of G and H on $k\times G$ by $g\cdot (x,g')= (x,gg')$ and $h\cdot (x,g')= (x,g'f_x(h)^{-1})$. These actions commute with each other, so we have an action of G on the quotient space $V:= (k\times G)/H$. Set $\varphi = \pi _{k\times G,H}$. Since H is finite, $\varphi$ is a geometric quotient. A straightforward calculation shows that for any $(x,g)\in k\times G$, the stabilizer $G_{\varphi (x,g)}$ is precisely $gK_xg^{-1}$. It follows that the G-orbits on V are all closed, but the assertion of Proposition 3.1 cannot hold for any$v\in V$. Hence no $v\in V$ admits an étale slice. Note that generic stabilizers are nontrivial, but there do exist orbits with trivial stabilizer (take $x= 0$).

Nonetheless we can even show (using étale slice methods!) that generic G-orbits in V are separable. Let $O= \{x\in k\mid x^2\ne 0, x, \ldots , (p-1)x\}$, an open subset of k. Then the finite group H acts freely on $O\times G$, so by [1, Proposition 8.2], $O\times G$ is a principal H-bundle in the étale topology in the sense of [1, Defn. 8.1]. Let $x\in O$. It follows that the derivative $d_{(x,g)}\varphi$ is surjective for all $g\in G$. Define an H-equivariant map $\psi _x:G\rightarrow k\times G$ by $\psi _x(g)= (x,g)$. An easy computation shows that the map $(\psi _x)_H:G/H\rightarrow V$ induced by $\psi _x$ is bijective and separable when regarded as a map onto its image, so $(\psi _x)_H$ gives by Zariski’s Main Theorem an isomorphism from G / H onto its image. Now $(\psi _x)_H$ is G-equivariant, where we let G act on G / H by left multiplication. Since $\pi _{G,H}:G\rightarrow G/H$ is separable, the orbit $G\cdot \pi _{G,H}(g)$ is separable for any $g\in G$. This means that the orbit $G\cdot \varphi (x,g)= G\cdot (\psi _x)_H(\pi _{G,H}(g))$ is separable as well.

In contrast, consider the orbit $G\cdot \varphi (0,g)$. This cannot be separable: for otherwise $\varphi (0,g)$ admits an étale slice by [1, Proposition 7.6], since the stabilizer $G_{\varphi (0,g)}$ is trivial, and we know already that this is impossible. It follows easily that $(\psi _0)_H:G/H\rightarrow V$ is not an isomorphism onto its image. We see from this that if i is the obvious inclusion of $Y:= \{0\}\times G$ in $k\times G$ then the induced map $i_G:Y/H\rightarrow (k\times G)/H= V$ is not an isomorphism onto its image (cf. Remark 2.4).

The failure of Proposition 3.1 and other consequences of the machinery of étale slices when slices do not exist is behind many of the technical difficulties we need to overcome in order to prove Theorem 1.1.

3.2 Some results on closed orbits

We first need a technical lemma which collects together various properties of orbits and quotients and the associated morphisms. For more details, see the proofs of [48, Lemmas 4.2, 10.1.3] or the discussion in [27, Sect.  2.1], for example; the extension to non-connected G is immediate. Note that if G acts on a variety X then for any $x\in X$, $G\cdot x$ is locally closed [10, Proposition 1.8], so it has the structure of a quasi-affine variety.

Lemma 3.3

Let X be a G-variety. Suppose $x \in X$, and let $\psi _x:G/G_x \rightarrow G\cdot x$ be the natural map. Then:

• (i) $\psi _x$ is a homeomorphism;

• (ii) $G\cdot x$ is affine if and only if $G/G_x$ is affine if and only if $G_x$ is reductive;

• (iii) $\psi _x$ is an isomorphism of varieties if and only if the orbit $G\cdot x$ is separable.

Remark 3.4

All the subtleties here are only really important in positive characteristic since in characteristic 0 the orbit map is always separable, so the morphism $\psi _x$ is always an isomorphism. The result shows that even in bad cases where the orbit map is not separable we can reasonably compare the quotient $G/G_x$ with the orbit $G\cdot x$, as one might hope.

Lemma 3.5

Let H be a subgroup of G and suppose $x \in X$. Set $K = G_x$ and let H act on X by restriction of the G-action. Then:

• (i) $H\cdot x$ is closed in $G\cdot x$ if and only if $HK = \{hk\mid h\in H,k\in K\}$ is a closed subset of G.

• (ii) If $G\cdot x$ is closed in X then $H\cdot x$ is closed in X if and only if HK is closed in G.

Proof

Part (ii) follows immediately from part (i). For part (i), since the map $\psi _x:G/K \rightarrow G\cdot x$ from Lemma 3.3 is a homeomorphism, $H\cdot x$ is closed in $G\cdot x$ if and only if the corresponding subset $H\cdot \pi _{G,K}(1)$ is closed in G / K (recall that $\pi _{G,K}:G\rightarrow G/K$ is the canonical projection). Since G / K has the quotient topology, this is the case if and only if the preimage of this orbit is closed in G. But the preimage is precisely the subset HK. $\square$

Our next result involves the following set-up: Suppose Y is another G-variety. Then $G\times G$ acts on the product $X\times Y$ via $(g_1,g_2)\cdot (x,y) = (g_1\cdot x,g_2\cdot y)$, and identifying G with its diagonal embedding $\Delta (G)$ in $G\times G$, we can also get the diagonal action of G on $X\times Y$: $g\cdot (x,y) = (g\cdot x,g\cdot y)$.

Lemma 3.6

With the notation just introduced, let $x \in X$, $y \in Y$ and set $K = G_x$, $H=G_y$. Then:

• (i) $H\cdot x$ is closed in $G\cdot x$ if and only if $K\cdot y$ is closed in $G\cdot y$ if and only if $G\cdot (x,y)$ is closed in $(G\cdot x)\times (G\cdot y)$.

• (ii) If $G\cdot x$ is closed in X and $G\cdot y$ is closed in Y, then $H\cdot x$ is closed in X if and only if $K\cdot y$ is closed in Y if and only if $G\cdot (x,y)$ is closed in $X \times Y$.

Proof

(i). The first equivalence follows from Lemma 3.5 since $KH = (HK)^{-1}$ is closed in G if and only if HK is closed in G (note that this argument is based on the one in the proof of [48, Lemma  10.1.4]). For the second equivalence, consider the orbit map $\kappa _1:G\times G \rightarrow G$ associated to the orbit of $1\in G$ for the double coset action of $G\times G$ on G (cf. Sect. 9); so $\kappa _1$ is given by $\kappa _1(g_1,g_2) = g_1g_2^{-1}$. Then $\kappa _1$ is surjective and open. Now, since the $(G\times G)$-orbit of (x, y) is $(G\cdot x) \times (G\cdot y)$ and the stabilizer of (x, y) in $G\times G$ is $K \times H$, we have that $G\cdot (x,y)= \Delta (G)\cdot (x,y)$ is closed in $(G\cdot x) \times (G\cdot y)$ if and only if $\Delta (G)(K\times H)$ is closed in $G\times G$, by Lemma 3.5(i). Now $\Delta (G)(K\times H)$ is closed in $G\times G$ if and only if $(K\times H)\Delta (G)$ is, and $(K\times H)\Delta (G) = \kappa _1^{-1}(KH)$. Since $\kappa _1$ is a surjective open map, we conclude that $\Delta (G)(K\times H)$ is closed in $G\times G$ if and only if KH is closed in G, which happens if and only if $K\cdot y$ is closed in $G\cdot y$, by Lemma 3.5(i) again.

(ii). This chain of equivalences follows quickly from part (i). $\square$

Remark 3.7

The results above give criteria for a result of the form “$G\cdot x$ closed implies $H\cdot x$ closed” for a point x in a G-variety X. We can’t hope for a general converse to this. For example, let G be any connected reductive group and, in the language of Sect. 2.4, let $x \in X=G^n$ be a generic tuple for a Borel subgroup of G and $y \in Y=G^n$ be a generic tuple for G itself. Then, $G_x = G_y = Z(G)$, the G-orbits of y and (x, y) are closed, but the G-orbit of x is not closed.

3.3 Finite morphisms and quotients

In this section we provide some general results on finite morphisms and quotients by reductive group actions. We begin with an extension of Zariski’s Main Theorem which deals with nonseparable morphisms. Recall that if X is an irreducible affine variety then $\nu _X:{\widetilde{X}}\rightarrow X$ denotes the normalization of X.

Proposition 3.8

Let $\phi :X\rightarrow Y$ be a dominant quasi-finite morphism of irreducible affine varieties. Suppose Y is normal and generic fibres of $\phi$ are singletons. Then $\phi$ is a finite bijection onto an open subvariety of Y. Moreover, the normalization map $\nu _X:{\widetilde{X}}\rightarrow X$ is a bijection.

Proof

As $\phi$ is dominant, we may identify k[Y] with a subring of k[X] and k(Y) with a subfield of k(X). The hypothesis on the fibres of $\phi$ implies that $\phi$ is purely inseparable [25, Theorem  4.6]. Let $f_1,\ldots , f_r$ be generators for k[X] as a k-algebra. Then there exists a power q of p such that $f_i^q\in k(Y)$ for all i. Let S be the k-algebra generated by k[Y] together with $f_1^q,\ldots , f_r^q$ and let Z be the corresponding affine variety, so that $S=k[Z]$. Then the inclusions $k[Y]\subseteq k[Z]\subseteq k[X]$ give rise to maps $\psi :X\rightarrow Z$ and $\alpha :Z\rightarrow Y$ such that $\phi = \alpha \circ \psi$. Now k[X] is integral over k[Z] by construction, so $\psi$ is finite and surjective, and hence $\alpha$ is quasi-finite and has the same image as $\phi$. But $\alpha$ is birational by construction, so $\alpha$ is an isomorphism from the affine variety Z onto an open subvariety of Y by Zariski’s Main Theorem (since Y is normal). To complete the proof of the first assertion, it is enough to show that $\psi$ is injective. This follows because any k-algebra homomorphism $k[X]\rightarrow k$ is completely determined by its values on $f_1^q,\ldots , f_r^q$, which are elements of k[Z].

Because $\nu _X$ is finite and birational, the map $\phi \circ \nu _X:{\widetilde{X}}\rightarrow Y$ satisfies the hypotheses of the proposition. Hence $\phi \circ \nu _X$ is injective. This forces $\nu _X$ to be injective also. But $\nu _X$ is also surjective, and we are done. $\square$

We need some further results about the behaviour of affine G-varieties under normalization. If X is an affine G-variety then ${\widetilde{X}}$ inherits a unique structure of a G-variety such that $\nu _X$ is G-equivariant (cf. [1, Sect.  3]). This gives a map of quotients $(\nu _X)_G:{\widetilde{X}}/G \rightarrow X/G$.

Lemma 3.9

Let X be an irreducible affine G-variety with good dimension and let $(\nu _X)_G$ be as above. Then $(\nu _X)_G$ is finite and ${\widetilde{X}}/G$ is the normalization of X / G.

Proof

The natural map of quotients $X/G^0 \rightarrow X/G$ can be viewed as the quotient map by the finite group $G/G^0$ and is therefore finite. The same is true for ${\widetilde{X}}/G^0 \rightarrow {\widetilde{X}}/G$, so by Remark 2.2(ii) it follows that $(\nu _X)_G$ is finite if $(\nu _X)_{G^0}$ is. Hence we may assume that G is connected.

The coordinate ring $k[{\widetilde{X}}]$ of the normalization of X is the integral closure of k[X] in the function field k(X). Let S be the integral closure of $k[X]^G$ in k(X). Then S is finitely generated as a k-algebra [1, 2.4.3], and $S\subseteq k[{\widetilde{X}}]$ as $k[{\widetilde{X}}]$ is integrally closed, so $S\subseteq k[{\widetilde{X}}]^G$ as G is connected (see the proof of [1, 2.4.1]). Let Z be the affine variety corresponding to S. Then $(\nu _X)_G$ factors as ${\widetilde{X}}/G{\mathop {\rightarrow }\limits ^{\alpha }} Z{\mathop {\rightarrow }\limits ^{\beta }} X/G$. It is clear that Z is normal (in fact, S is the integral closure of $k[X]^G$ in $k(X)^G$, so Z is the normalization of X / G). Now $(\nu _X)_G$ is birational and quasi-finite by Lemmas 2.9 and 2.8(ii), so $\alpha$ is also birational and quasi-finite. It follows from Zariski’s Main Theorem that $\alpha$ is an open embedding.

The map $\beta$ is finite by construction, so to complete the proof that $(\nu _X)_G$ is finite it is enough to show that $\alpha$ is surjective. Define $\theta :{\widetilde{X}}\rightarrow Z\times X$ by $\theta = (\alpha \circ \pi _{{\widetilde{X}},G})\times \nu _X$ and let C be the closure of $\theta ({\widetilde{X}})$. We have a commutative diagram

where ${\mathrm{pr}}_1$ is projection onto the first factor. The composition ${\widetilde{X}}\rightarrow C\rightarrow X$ is finite, where the second map is projection onto the second factor, so $\theta$ is a finite map from ${\widetilde{X}}$ to C; in particular, $C= \theta ({\widetilde{X}})$.

Let G act on $Z\times X$ trivially on the first factor, and by the given action on the second. It is immediate that $\theta$ is G-equivariant, so C is G-stable and we have an induced map $\theta _G:{\widetilde{X}}/G\rightarrow (Z\times X)/G$. The image D of $\theta _G$ is $\pi _{Z\times X,G}(C)$, and this is closed in $(Z\times X)/G$ as C is closed and G-stable. There is an obvious map $\xi :(Z\times X)/G\rightarrow Z\times X/G$, and it is easily checked that $\xi$ is an isomorphism; hence $\xi (D)$ is closed. Untangling the definitions, we find that $\alpha$ factors as ${\widetilde{X}}/G{\mathop {\rightarrow }\limits ^{\theta _G}} (Z\times X)/G{\mathop {\rightarrow }\limits ^{\xi }} Z\times X/G{\mathop {\rightarrow }\limits ^{\tau }} Z$, where $\tau$ is projection onto the first factor.

Clearly $\xi (D)$ is contained in the subset $\{(z,e)\in Z\times X/G\mid \beta (z)= e\}$, which we can identify with Z via $\tau$. It follows that $\alpha ({\widetilde{X}}/G)= \tau (\xi (D))$ is closed in Z. But $\alpha ({\widetilde{X}}/G)$ is a nonempty open subset of Z, so we must have $\alpha ({\widetilde{X}}/G)= Z$, as required.

To finish the proof, we note that for any G (connected or otherwise), the variety ${\widetilde{X}}/G$ is normal since ${\widetilde{X}}$ is normal, and the considerations above show that $(\nu _X)_G:{\widetilde{X}}/G \rightarrow X/G$ is finite. Moreover, $(\nu _X)_G$ is birational by Lemma 2.9 since X has good dimension. The result now follows from another application of Zariski’s Main Theorem. $\square$

Next we extend a result of Bardsley and Richardson [1, 2.4.2], which they prove in the special case when X and Y are normal and $\phi$ is dominant. It provides an extension to positive characteristic of a result used freely in [33].

Proposition 3.10

Let $\phi :X\rightarrow Y$ be a finite G-equivariant morphism of affine G-varieties. Then $\phi _G:X/G\rightarrow Y/G$ is finite.

Proof

As at the start of the proof of Lemma 3.9, we can immediately reduce to the case when G is connected, since the natural maps $X/G^0\rightarrow X/G$ and $Y/G^0 \rightarrow Y/G$ are finite. The map $X_{\mathrm{cl}}/G\rightarrow X/G$ is surjective, and Lemma 2.3 implies it is finite. We may also assume, therefore, that X has good dimension. Since a morphism is finite if and only if its restriction to every irreducible component of the domain is finite, we can assume X is irreducible. By the proof of Lemma 2.8, $\phi (X_{\mathrm{cl}})\subseteq Y_{\mathrm{cl}}$, so after replacing Y with $\overline{\phi (X)}$ if necessary, we may assume by Lemma 2.3 that $\phi$ is dominant and Y is irreducible and has good dimension.

The map $\phi :X\rightarrow Y$ gives rise to a map ${\widetilde{\phi }}:{\widetilde{X}}\rightarrow {\widetilde{Y}}$, and ${\widetilde{\phi }}$ is finite as $\phi$ is. We have a commutative diagram

where the vertical arrows are the normalization maps. Taking quotients by G, we obtain a commutative diagram

Since ${\widetilde{\phi }}$ is finite and dominant and ${\widetilde{X}}$ and ${\widetilde{Y}}$ are irreducible and normal, the map ${\widetilde{\phi }}_G:{\widetilde{X}}/G\rightarrow {\widetilde{Y}}/G$ is finite and dominant [1, 2.4.2]. Now Lemma 3.9 shows that the map $(\nu _Y)_G:{\widetilde{Y}}/G \rightarrow Y/G$ is finite and so $(\nu _Y)_G\circ {\widetilde{\phi }}_G$ is finite. Therefore, $\phi _G\circ (\nu _X)_G$ is finite and by Remark 2.2(ii) we get that $\phi _G$ is finite, as required. $\square$

4 Proof of Theorem 1.1, Part 1: quasi-finiteness

In this section we provide the first step towards our proof of Theorem 1.1, showing that the map $\psi _{X,H}$ in question is quasi-finite. We are also able to retrieve other results from [33] which follow from the main theorem, but in arbitrary characteristic. Our first result is a generalization of [2, Theorem  4.4]; see also [7, Theorem  5.4].

Proposition 4.1

Suppose that G is a reductive group and X is an affine G-variety. Let H be a G-completely reducible subgroup of G and let $x\in X^H$. Then the following are equivalent:

• (i) $N_G(H)\cdot x$ is closed in X;

• (ii) $G\cdot x$ is closed in X and H is $G_x$-cr.

Proof

First suppose $G\cdot x$ is not closed. Let P(x) and $\Omega (x)$ be the R-parabolic subgroup and class of cocharacters given by Theorem 2.12. Since $H \le G_x \le P(x)$ is G-cr, there exists an R-Levi subgroup L of P(x) containing H. Since $R_u(P(x))$ acts simply transitively on $\Omega (x)$ and on the set of R-Levi subgroups of P(x), there exists $\lambda \in \Omega (x)$ with $L = L_\lambda$. But then $H \subseteq L_\lambda$ means that $\lambda \in Y(C_G(H)) \subseteq Y(N_G(H))$; in particular, $\lambda (a)\cdot x \in N_G(H)\cdot x$ for all $a \in k^*$. Now $\lim _{a\rightarrow 0} \lambda (a)\cdot x$ exists in X and is not G-conjugate to x, so it is not $N_G(H)$-conjugate to x, so $N_G(H)\cdot x$ is not closed. This shows that if (i) holds then $G\cdot x$ must be closed. Therefore, in order to finish the proof, we need to show that $N_G(H)\cdot x$ is closed if and only if H is $G_x$-cr under the assumption that $G\cdot x$ is closed (note that since $G\cdot x$ is closed, $G_x$ is reductive (Lemma 3.3(ii)), and hence it makes sense to ask whether or not H is $G_x$-cr).

To see this equivalence, let ${\mathbf {h}} \in G^n$ for some n be a generic tuple for the subgroup H and consider the diagonal action of G on $G^n\times X$. Then $C_G(H) = G_{{\mathbf {h}}}$. Now, by Lemma 3.6, since $G\cdot x$ is closed in X, $C_G(H)\cdot x$ is closed in X if and only if $G_x\cdot {\mathbf {h}}$ is closed in $G^n$. The latter condition is equivalent to requiring that H is $G_x$-cr, and since x is H-fixed and $N_G(H)$ is a finite extension of $HC_G(H)$, $C_G(H)\cdot x$ is closed in X if and only if $N_G(H)\cdot x$ is closed in X. This completes the proof. $\square$

Remarks 4.2

(i). In characteristic 0, the subgroup H of G is G-cr if and only if H is reductive. In this case, therefore, we are just requiring that H is reductive and the second condition in part (ii) of the Theorem is then automatic. Therefore, when ${{\,\mathrm{char}\,}}(k) = 0$, we retrieve Luna’s result [33, Sect. 3, Cor. 1].

(ii). The implication (ii) implies (i) of Proposition 4.1 is not true in general without the hypothesis that H is $G_x$-cr, as a straightforward modification of [2, Ex. 4.6] shows. See also [6, Ex. 5.1, Ex. 5.3], noting that if A, B are commuting G-cr subgroups of G and B is not $C_G(A)$-cr then B is not $N_G(A)$-cr by [6, Proposition 2.8].

(iii) Suppose H is a torus in Proposition 4.1; then H is linearly reductive, so H is G-cr. Now $N_G(H)$ is a finite extension of the Levi subgroup $C_G(H)$ of G, so $N_G(H)\cdot x$ is closed if and only if $C_G(H)\cdot x$ is closed. Moreover, H is automatically $G_x$-cr if $G_x$ is reductive. It follows that $G\cdot x$ is closed if and only if $C_G(H)\cdot x$ is closed. (This is also a special case of [4, Theorem 5.4].) We use this result repeatedly in Sect. 9.

Some of the constructions used in the proof of the next result are based on those in [8, Sec. 3.8].

Lemma 4.3

Suppose H is a reductive subgroup of G such that H is not G-cr. Then:

• (i) There exists an affine G-variety X and a point $x \in X^H$ such that $G\cdot x$ is not closed.

• (ii) There exists a rational G-module V and a nonzero subspace $W\subseteq V^H$ such that:

  • (a) 0 lies in the closure of $G\cdot w$ for all $w\in W$;

  • (b) $N_G(H)\cdot w$ is finite (hence closed in V) for all $w\in W$.

  In particular, if $N_G(H)$ is reductive, then the map $\psi _{V,H}:V^H/N_G(H) \rightarrow V/G$ is not quasi-finite.

Proof

Choose a closed embedding $G \hookrightarrow {{\,\mathrm{SL}\,}}_m(k)$ for some m and think of H and G as closed subgroups of ${{\,\mathrm{SL}\,}}_m(k)$. Let ${{\,\mathrm{Mat}\,}}_m$ denote the algebra of all $m\times m$ matrices. Let ${\mathbf {x}} = (x_1,\ldots ,x_n) \in H^n$ be a basis for the associative subalgebra of ${{\,\mathrm{Mat}\,}}_m$ spanned by H; then ${\mathbf {x}}$ is a generic tuple for H (see Sect. 2.4). This means that if we let ${{\,\mathrm{SL}\,}}_m(k)$ act on $Y := ({{\,\mathrm{Mat}\,}}_m)^n$ by simultaneous conjugation, then $G\cdot {\mathbf {x}}$ is not closed. Note that since H is itself H-cr, the H-orbit of $H\cdot {\mathbf {x}}$ is closed in Y.

There is also a right action of ${{\,\mathrm{GL}\,}}_n(k)$ on Y, which we denote by $*$. Given a matrix $A=(a_{ij}) \in {{\,\mathrm{GL}\,}}_n(k)$ and an element ${\mathbf {y}} = (y_1,\ldots ,y_n) \in Y$, we can set

$$\begin{aligned} {\mathbf {y}}*A = \left( \sum _{i=1}^n a_{i1}y_i, \ldots , \sum _{i=1}^n a_{in}y_i\right) . \end{aligned}$$

This is the action obtained by thinking of the tuple ${\mathbf {y}}$ as a row vector of length n and letting the $n\times n$ matrix A act on the right in the obvious way. Note that the ${{\,\mathrm{SL}\,}}_m(k)$- and ${{\,\mathrm{GL}\,}}_n(k)$-actions commute.

Given any $h \in H$, since ${\mathbf {x}}$ is a basis for the associative algebra generated by H, we have that $h\cdot {\mathbf {x}}$ is also a basis for this algebra, and hence there exists a unique $A(h) \in {{\,\mathrm{GL}\,}}_n(k)$ such that $h \cdot {\mathbf {x}} = {\mathbf {x}} *A(h)$. Note also that

$$\begin{aligned} (h_1h_2) \cdot {\mathbf {x}} = h_1\cdot (h_2\cdot {\mathbf {x}}) = h_1\cdot ({\mathbf {x}}*A(h_2)) = (h_1\cdot {\mathbf {x}})*A(h_2) = {\mathbf {x}} *(A(h_1)A(h_2)), \end{aligned}$$

and hence the map $A:H \rightarrow {{\,\mathrm{GL}\,}}_n(k)$ is a group homomorphism. This map is in fact a rational representation of H since it arises from the morphic action of H on the vector space spanned by the entries of ${\mathbf {x}}$. Let K denote the image of H in ${{\,\mathrm{GL}\,}}_n(k)$; then K is a reductive group and ${\mathbf {x}}*K= H\cdot {\mathbf {x}}$ is closed. Moreover, since the elements of the tuple ${\mathbf {x}}$ are linearly independent, the stabilizer of ${\mathbf {x}}$ in K is trivial. Hence ${\mathbf {x}}$ is a stable point for the action of K on Y. Now let $X= Y/K$ and set $x:=\pi _{Y,K}({\mathbf {x}})$. Since the ${{\,\mathrm{SL}\,}}_m(k)$- and ${{\,\mathrm{GL}\,}}_n(k)$-actions on Y commute, we obtain an action of ${{\,\mathrm{SL}\,}}_m(k)$ on X. It is immediate that $x \in X^H$.

We know that $G\cdot {\mathbf {x}}$ is not closed in Y, so there exists a cocharacter $\lambda \in Y(G)$ such that $\lim _{a\rightarrow 0} \lambda (a) \cdot {\mathbf {x}} = {\mathbf {y}}$ exists and is not G-conjugate to ${\mathbf {x}}$. Since $\pi _{Y,K}$ is G-equivariant, it is easy to see that $\lim _{a\rightarrow 0} \lambda (a) \cdot x = \pi _{Y,K}({\mathbf {y}})$ (and in particular this limit exists). Suppose $\pi _{Y,K}({\mathbf {y}})$ is G-conjugate to x. Then there exists $g \in G$ such that $g\cdot \pi _{Y,K}({\mathbf {y}}) = \pi _{Y,K}(g\cdot {\mathbf {y}}) = x$, so $g\cdot {\mathbf {y}} \in \pi _{Y,K}^{-1}(x) = \pi _{Y,K}^{-1}(\pi _{Y,K}({\mathbf {x}}))$. But ${\mathbf {x}}$ is a stable point for K, so $\pi _{Y,K}^{-1}(\pi _{Y,K}({\mathbf {x}}))$ is precisely $K\cdot {\mathbf {x}}$, which coincides with $H\cdot {\mathbf {x}}$ by construction. Hence $g\cdot {\mathbf {y}} = h\cdot {\mathbf {x}}$ for some $h \in H$ and we see that ${\mathbf {y}}$ and ${\mathbf {x}}$ are G-conjugate, which is a contradiction. Hence $\pi ({\mathbf {y}})$ and x are not conjugate, and the G-orbit of $x \in X^H$ is not closed, which proves (i).

To prove (ii), let S denote the unique closed G-orbit in the closure of $G\cdot x$. Then, following [28, Lemma 1.1(b)], we can find a rational G-module V with a G-equivariant morphism $\phi :X\rightarrow V$ such that $\phi ^{-1}(0) = S$. Since $G\cdot x$ is not closed, it does not meet S, and hence $v:=\phi (x) \ne 0$. However, by Theorem 2.12, there exists $\mu \in Y(G)$ such that $\lim _{a\rightarrow 0} \mu (a)\cdot x \in S$, and since the morphism $\phi$ is G-equivariant, we have that $\{0\}$ is the unique closed G-orbit in the closure of $G\cdot v$. Note also that v is H-fixed since x is. Now the tuple ${\mathbf {x}}$ consists of elements of H, so is $C_G(H)$-fixed, and hence $x = \pi _{Y,K}({\mathbf {x}})$ is also $C_G(H)$-fixed, which means that x is actually $HC_G(H)$-fixed. Since H is reductive, $N_G(H)^0 = H^0C_G(H)^0$, so x is $N_G(H)^0$-fixed and hence the $N_G(H)$-orbit of x is finite. This in turn implies that the $N_G(H)$-orbit of v is finite, and hence closed in V. Finally, let $W\subseteq V^H$ be the one-dimensional subspace of V spanned by v. Then for all $w \in W$, $N_G(H)\cdot w$ is finite, hence closed, and 0 is in the closure of $G\cdot w$, so we have parts (a) and (b) of (ii). For the second statement, if $N_G(H)$ is reductive—so that it definitely makes sense to talk about the quotient $V^H/N_G(H)$—then the image of W in $V^H/N_G(H)$ is still infinite, but every element of this infinite set is mapped to the point corresponding to 0 in V / G under the natural morphism $V^H/N_G(H) \rightarrow V/G$, so this morphism cannot be quasi-finite. $\square$

With this result in hand, we can provide the first step towards the proof of Theorem 1.1 by showing that the morphism $\psi _{X,H}$ is quasi-finite.

Theorem 4.4

Suppose H is a reductive subgroup of G. The following conditions on H are equivalent:

• (i) $N_G(H)$ is reductive and for every affine G-variety X, the natural morphism $\psi _{X,H}:X^H/N_G(H)\rightarrow X/G$ is quasi-finite;

• (ii) H is G-cr.

Proof

Suppose H is not G-cr. Then either $N_G(H)$ is not reductive, in which case the first part of condition (i) fails, or else $N_G(H)$ is reductive but the second part of condition (i) fails by Lemma 4.3(ii). Hence (i) implies (ii).

Conversely, suppose H is G-cr, and let X be any affine G-variety. Since H is G-cr, and hence H is reductive, we have $N_G(H)^0 = H^0C_G(H)^0$. That $N_G(H)$ is reductive is shown in [5, Proposition 3.12], and hence it always makes sense to take the quotient $X^H/N_G(H)$.

Suppose $x\in X^H$. We first claim that the unique closed G-orbit S in $\overline{G\cdot x}$ meets $X^H$. Indeed, either $G\cdot x$ is already closed, in which case $S=G\cdot x$, or we can find the optimal parabolic P(x) and optimal class $\Omega (x)$ as given in Kempf’s Theorem 2.12. Since $H\le G_x \le P(x)$ and H is G-cr, there is a Levi subgroup L of P(x) containing H. Since the unipotent radical acts simply transitively on $\Omega (x)$ and on the set of Levi subgroups of P(x), there is precisely one element $\lambda \in \Omega (x)$ with $L = L_\lambda$, and this choice of $\lambda$ commutes with H. But then $y:= \lim _{a\rightarrow 0}\lambda (a)\cdot x \in S\cap X^H$, which proves the claim.

Now any point of X / G has the form $\pi _{X,G}(x)$, where $G\cdot x$ is closed in X. So let $x\in X$ such that $G\cdot x$ is closed. For any $y\in \pi _{X,G}^{-1}(\pi _{X,G}(x)) \cap X^H$, $G\cdot x$ is the unique closed G-orbit in $\overline{G\cdot y}$. Hence, if $\pi _{X,G}^{-1}(\pi _{X,G}(x)) \cap X^H$ is nonempty, $G\cdot x$ must meet $X^H$, by the claim in the previous paragraph. It follows from the definitions that $\pi _{X^H,N_G(H)}^{-1}(\psi _{X,H}^{-1}(\pi _{X,G}(x))) = \pi _{X,G}^{-1}(\pi _{X,G}(x))\cap X^H$, so to show that $\psi _{X,H}$ is quasi-finite, we need to show that for each such x there are only finitely many closed $N_G(H)$-orbits in $\pi _{X,G}^{-1}(\pi _{X,G}(x))\cap X^H$. But any $y\in X^H$ with a closed $N_G(H)$-orbit has a closed G-orbit, by Proposition 4.1, and hence any $y\in \pi _{X,G}^{-1}(\pi _{X,G}(x))\cap X^H$ with a closed $N_G(H)$-orbit is already G-conjugate to x. So we must show that there are only finitely many closed $N_G(H)$-orbits in $G\cdot x\cap X^H$.

Fix $x\in X^H$ with $G\cdot x$ closed, and recall that $G_x$ is reductive since $G\cdot x$ is closed. Let $y\in G\cdot x\cap X^H$, and write $y=g\cdot x$ for some $g\in G$. Since $G\cdot y$ is closed, Proposition 4.1 says that $N_G(H)\cdot y$ is closed if and only if H is $G_y$-cr, which is the case if and only if $g^{-1}Hg$ is $G_x$-cr. Suppose $g^{-1}Hg$ and H are $G_x$-conjugate: say $H = g_1^{-1}(g^{-1}Hg)g_1$ for some $g_1\in G_x$. Then $gg_1 \in N_G(H)$ and $y = g\cdot x = (gg_1)\cdot x$, so we see that x and y are $N_G(H)$-conjugate. Conversely, suppose x and y are $N_G(H)$-conjugate: say $y= m\cdot x$ for some $m\in N_G(H)$. Then $m^{-1}g\in G_x$ and $m^{-1}g(g^{-1}Hg)g^{-1}m= H$, so $g^{-1}Hg$ and H are $G_x$-conjugate. Hence the distinct closed $N_G(H)$-orbits in $G\cdot x\cap X^H$ correspond to the distinct $G_x$-conjugacy classes of $G_x$-cr subgroups of the form $g^{-1}Hg$ inside $G_x$. It is therefore enough to show that there are only finitely many such conjugacy classes.

Let ${\mathbf {h}} \in H^n$ be a generic tuple for H in $G_x$ for some n and let $g\in G$ such that $g^{-1}Hg$ is a $G_x$-cr subgroup of $G_x$. Then $g^{-1}\cdot {\mathbf {h}}$ is a generic tuple for $g^{-1}H g$. Since $g^{-1}Hg$ is both G-cr and $G_x$-cr, the G- and $G_x$-orbits of ${\mathbf {h}}$ in $G^n$ are both closed. It follows from [35, Theorem  1.1] that the natural map of quotients $G_x^n/G_x \rightarrow G^n/G$ is finite, and hence there are only finitely many closed $G_x$-orbits contained in $G\cdot {\mathbf {h}} \cap G_x^n$. This proves the result. $\square$

Remark 4.5

Note that if $G_x = H$ and $G\cdot x$ is closed then the argument in the proof above shows that there is precisely one closed $N_G(H)$-orbit inside $G\cdot x \cap X^H$ (namely, $N_G(H)\cdot x$), and therefore $\psi _{X,H}^{-1}(\pi _{X,G}(x))$ is a singleton. We will use this observation in Sects. 6 and 7.

The third paragraph of the proof above shows that for any $x \in X^H$, the unique closed orbit contained in $\overline{G\cdot x}$ also meets $X^H$. This allows us to prove the following:

Lemma 4.6

The map $\psi _{X,H}:X^H/N_G(H)\rightarrow X/G$ of Theorem 1.1 has closed image if and only if for all $x\in \overline{G\cdot X^H}$ such that $G\cdot x$ is closed, $x\in G\cdot X^H$.

Proof

Since $\overline{G\cdot X^H}$ is closed and G-stable, we may replace X with $\overline{G\cdot X^H}$; then saying $\psi _{X,H}$ has closed image is the same as saying that $\psi _{X,H}$ is surjective. But this is equivalent to saying that the fibre above every point of X / G meets $X^H$. Since each fibre contains a unique closed orbit, the observation before the Lemma gives the result. $\square$

Now we extend Luna’s result [33, Cor. 3] to positive characteristic.

Proposition 4.7

Suppose H is a reductive subgroup of G. The following conditions on H are equivalent:

• (i) for every affine G-variety X, every G-orbit in X that meets $X^H$ is closed;

• (ii) H is G-cr and $N_G(H)/H$ is a finite group.

Proof

Suppose (i) holds. Then H must be G-cr, by Lemma 4.3. Since H is reductive, $N_G(H)^0 = H^0C_G(H)^0$. Let $x \in C_G(H)^0$ and let G act on itself by conjugation. We have $x \in C_G(H) = G^H$, so the G-orbit of x (i.e., the conjugacy class of x) must be closed in G. As x belongs to $G^0$, it follows from [55, Cor. 3.6] that x is a semisimple element of G. Since $C_G(H)^0$ consists entirely of semisimple elements, it must be a torus [10, Cor. 11.5(1)]. Hence $N_G(H)^0 = H^0C_G(H)^0$ is a reductive group and $(N_G(H)/H)^0$ is a torus.

Now suppose, for contradiction, that $N_G(H)/H$ is infinite. Then there exists a one-dimensional subtorus S of $C_G(H)^0$ not contained in H. To ease notation, let $Z = HS$ and note that Z is reductive. Since H is normal in Z and $Z/H \cong S/(S\cap H)$ is a one-dimensional torus, we have a multiplicative character $\chi :Z \rightarrow k^*$ with kernel H; let V denote the corresponding 1-dimensional Z-module. Set $Y = G\times V$, let Z act on Y via $z\cdot (g,v) := (gz^{-1},\chi (z)v)$, and let G act by left multiplication on the first factor and trivially on the second factor. Now let $X = Y/Z$; this is a special case of a construction described in [32, I.3]. Since Z is reductive and Y is affine, X is affine, and since Z acts freely on Y, the fibres of $\pi _{Y,Z}$ are precisely the Z-orbits in Y. Moreover, since the G- and Z-actions on Y commute, X is naturally a G-variety. Let $0\ne v\in V$ and choose a cocharacter $\lambda$ of Z such that $m:= -\langle \lambda ,\chi \rangle > 0$. Then $\lambda (a)\cdot \pi _{Y,Z}(1,v)= \pi _{Y,Z}(1,\chi (\lambda (a^{-1}))v)= \pi _{Y,Z}(1,a^mv)$ for all $a\in k^*$, so $\lim _{a\rightarrow 0} \lambda (a)\cdot \pi _{Y,Z}(1,v)= \pi _{Y,Z}(1,0)\not \in G\cdot \pi _{Y,Z}(1,v)$, so $G\cdot \pi _{Y,Z}(1,v)$ is not closed. However, $\pi _{Y,Z}(1,v)$ is H-fixed, and we have our contradiction. Hence $N_G(H)/H$ is finite. This completes the proof that (i) implies (ii).

Conversely, suppose (ii) holds and X is any affine G-variety. Let $x \in X^H$, so that $H \le G_x$. Since $N_G(H)/H$ is finite, $N_G(H)\cdot x$ is a finite union of H-orbits. But $N_G(H)\cdot x \subseteq X^H$, so each of these H-orbits is a singleton and $N_G(H)\cdot x$ is finite, and therefore closed in X. Now we can apply Proposition 4.1 to deduce that the G-orbit of x is also closed, which gives (i). $\square$

5 Proof of Theorem 1.1, Part 2: surjectivity

In this section, we prove the following:

Theorem 5.1

Let X be an affine G-variety and let H be a G-cr subgroup of G. Then the map $\psi _{X,H}:X^H/N_G(H)\rightarrow X/G$ has closed image.

The proof of Theorem 5.1 in positive characteristic requires some preparation. Before we begin, we note that if $\mathrm{char}(k)= 0$ then we can give a much quicker proof using the machinery of étale slices, as follows. Let $x\in \overline{G\cdot X^H}$ such that $G\cdot x$ is closed. Then there is an étale slice through x for the G-action [32, III.1]. By Proposition 3.1, there is an open G-stable neighbourhood O of x such that $G_y$ is conjugate to a subgroup of $G_x$ for all $y\in O$. Since O meets $G\cdot X^H$, H must be conjugate to a subgroup of $G_x$. Hence $x\in G\cdot X^H$, and we are done by Lemma 4.6.

We need some material on weighted projective varieties and their quotients by reductive groups (cf. [44, Chapter 3, Sect. 4]). Let V be a G-module equipped with an action of $k^*$ which commutes with the action of G. Suppose the weights of $k^*$ on V are all positive, so that $\lim _{c\rightarrow 0} c\cdot v = 0$ for every $v\in V$, where the $c\in k^*$. The action of $k^*$ decomposes V into weight spaces, and this in turn gives a grading by non-negative integers of the coordinate ring k[V]. Let $k[V]_i$ denote the $i{\mathrm{th}}$-graded piece of k[V]. We say that $f\in k[V]$ is homogeneous if $f\in k[V]_i$ for some i; in this case we write $\mathrm{deg}(f)= i$ (so $\mathrm{deg}(f)$ is the weighted degree rather than the usual degree of a polynomial). The action of $k^*$ can be diagonalised, so we can choose a basis $\{v_1,\ldots , v_n\}$ for V consisting of weight vectors. Then the corresponding elements $X_1,\ldots , X_n$ of the dual $V^*$ are weight vectors and we can write $k[V] = k[X_1,\ldots ,X_n]$; we set $d_i= \mathrm{deg}(X_i)$ for each i.

Set ${{\mathbb {W}}}(V) = \mathrm {Proj}(k[V])$; we call this the weighted projectivization of V according to the $k^*$-action [18]. Then ${{\mathbb {W}}}(V)$ is a projective variety and we may identify the points of ${{\mathbb {W}}}(V)$ with the equivalence classes of $V{\setminus } \{0\}$ under the equivalence relation $\sim$, where $v\sim w$ if and only if $v= c\cdot w$ for some $c\in k^*$. If the weights of the $k^*$-action on V are all 1—that is, if the action of $k^*$ on V is by ordinary scalar multiplication—then ${{\mathbb {W}}}(V)$ is just the ordinary projective space ${{\mathbb {P}}}(V)$ associated to V, but in Sect. 6 we will need to consider the general weighted case. One can show that the canonical projection $\xi _V:V\backslash \{0\}\rightarrow {{\mathbb {W}}}(V)$ is a good quotient. If $f\in k[V]$ is homogeneous and $\mathrm{deg}(f)\ge 1$ then we set ${{\mathbb {W}}}(V)_f= \{\xi _V(v)\mid v\in V, f(v)\ne 0\}$; then ${{\mathbb {W}}}(V)_f$ is an open affine subset of ${{\mathbb {W}}}(V)$, with coordinate ring $(k[V]_f)_0$ (the zero-graded part of the localisation $k[V]_f$).

Since the G- and $k^*$-actions commute, the ring $k[V]^G$ of invariants also inherits a grading by non-negative integers: if $f \in k[V]^G$ and $f = f_0 + \cdots +f_r$ is a decomposition with $f_i \in k[V]_i$ for each i, then $f_i \in (k[V]^G)_i$ for each i. It is easily checked that the action of G on V descends to give an action of G on ${{\mathbb {W}}}(V)$. We say that $x\in {{\mathbb {W}}}(V)$ is a semistable point (or G-semistable point) if $x\in {{\mathbb {W}}}(V)_f$ for some homogeneous $f\in k[V]^G$ such that $\mathrm{deg}(f)\ge 1$; otherwise we say that x is unstable (or G-unstable). We define ${{\mathbb {W}}}(V)_{\mathrm{ss}, G}$ to be the set of G-semistable points of ${{\mathbb {W}}}(V)$; this is an open subset of ${{\mathbb {W}}}(V)$.

Let $Y= \mathrm{Proj}(k[V]^G)$. Then Y is a projective variety and the inclusion of $k[V]^G$ in k[V] gives rise to a map $\eta _{V,G}:{{\mathbb {W}}}(V)_{\mathrm{ss},G}\rightarrow Y$. It follows from the proof of [44, Theorem  3.14] that Y is a good quotient of ${{\mathbb {W}}}(V)_{\mathrm{ss},G}$ in the sense of [44, Chapter 3, Sect. 4, p 57] (the argument given in loc. cit. is only for the ordinary projective variety ${{\mathbb {P}}}(V)$, but it is clear that it holds for the weighted case as well). We set ${{\mathbb {W}}}(V)_{\mathrm{ss}, G}/G:= Y$. Moreover, if $f\in k[V]^G$ is homogeneous and $\mathrm{deg}(f)\ge 1$ then $Y_f:= \eta _{V,G}({{\mathbb {W}}}(V)_f)$ is an open affine subvariety of Y, with coordinate ring $((k[V]^G)_f)_0$, and the induced map of affine varieties from ${{\mathbb {W}}}(V)_f$ to $Y_f$ is a good quotient.

We have an analogous notion of semistable points in the affine variety V. We say that $v\in V$semistable (or G-semistable) if $f(v)\ne 0$ for some homogeneous $f\in k[V]^G$ such that $\mathrm{deg}(f)\ge 1$, and we define $V_{\mathrm{ss},G}$ to be the set of semistable points; note that $V_{\mathrm{ss},G}= \xi _V^{-1}({{\mathbb {W}}}(V)_{\mathrm{ss},G})$. If v is not stable then we say that v is unstable (or G-unstable). Since the homogeneous elements of $k[V]^G$ generate $k[V]^G$, v is unstable if and only if $\pi _{V,G}(v)= \pi _{V,G}(0)$. By the Hilbert–Mumford Theorem, this is the case if and only if there exists $\lambda \in Y(G)$ such that $\lim _{a\rightarrow 0} \lambda (a)\cdot v= 0$. We denote the composition $V_{\mathrm{ss},G}\rightarrow {{\mathbb {W}}}(V)_{\mathrm{ss},G}{\mathop {\longrightarrow }\limits ^{\eta _{V,G}}} Y$ by $\nu _{V,G}$.

Now suppose K is a reductive subgroup of G and X is a closed $(K\times k^*)$-stable subvariety of V (so that in particular $0 \in X$). Then the vanishing ideal for X in k[V] is homogeneous with respect to our fixed $k^*$-grading, so k[X] inherits a grading. The constructions above still go through replacing V and G with X and K. We have projective varieties ${{\mathbb {W}}}(X):= \mathrm{Proj}(k[X])$ and ${{\mathbb {W}}}(X)_{\mathrm{ss}, K}/K:= Z:= \mathrm{Proj}(k[X]^K)$, where ${{\mathbb {W}}}(X)_{\mathrm{ss},K}$ is defined analogously to above; the map ${{\mathbb {W}}}(X)_{\mathrm{ss},K}\rightarrow Z$ is a good quotient. Note that the proof of [44, Theorem  3.14] still goes through: all one needs is that k[X] is graded and the G-action preserves the grading. Since ${{\mathbb {W}}}(V)$ and ${{\mathbb {W}}}(X)$ are categorical quotients of $V{\setminus } \{0\}$ and $X{\setminus } \{0\}$ respectively, the inclusion of X in V gives rise to a map from ${{\mathbb {W}}}(X)$ to ${{\mathbb {W}}}(V)$.

It is clear from the characterisation of semistable points in terms of the Hilbert–Mumford Theorem that $X\cap V_{\mathrm{ss},G}\subseteq X_{\mathrm{ss},K}$. Suppose $X_{\mathrm{ss},K}\subseteq V_{\mathrm{ss},G}$; then $X_{\mathrm{ss},K}= X\cap V_{\mathrm{ss},G}$. Since Y and Z are categorical quotients of ${{\mathbb {W}}}(V)_{\mathrm{ss},G}$ and ${{\mathbb {W}}}(X)_{\mathrm{ss},K}$, respectively, the inclusion of ${{\mathbb {W}}}(X)_{\mathrm{ss},K}$ in ${{\mathbb {W}}}(V)_{\mathrm{ss},G}$ gives rise to a map $\phi :Z\rightarrow Y$. Now we come to the point: because Y and Z are projective, the image of $\phi$ is closed.

We can now state and prove the main result of this section.

Proposition 5.2

Let V be a G-module equipped with a $k^*$-action as above. Let K be a reductive subgroup of G, let X be a closed $(K\times k^*)$-stable subset of V and suppose $X_{\mathrm{ss},K}\subseteq V_{\mathrm{ss},G}$. Then the natural morphism of quotients $X/K \rightarrow V/G$ has closed image (i.e., $\pi _{V,G}(X)$ is closed in V / G).

Proof

For the purposes of the proof, we need to replace G with a slightly larger group to take into account possible effects of passing to the weighted projectivisation. Without loss, we may assume that G is a subgroup of ${{\,\mathrm{GL}\,}}(V)$. Let $R = k[V]^G$ and let $f_1,\ldots , f_r$ be homogeneous generators for R. Let m be the lowest common multiple of the degrees $\deg (f_1),\ldots ,\deg (f_r)$ and write $m = p^\alpha m'$ for some $m'$ coprime to p. Let F be the finite group of $m'^{\mathrm{th}}$ roots of unity, regarded as a subgroup of $k^*$ (equipped with its given action on V). Now set $\Gamma = FG$. Then $\Gamma$ inherits an action on V from the commuting actions of G and $k^*$, and $V_{\mathrm{ss},\Gamma } = V_{\mathrm{ss},G}$ because $\Gamma ^0 = G^0$. Further, F acts on the quotient V / G and the quotient map $\pi _{V/G,F}:V/G \rightarrow (V/G)/F = V/\Gamma$ is a geometric quotient. The subset X of V is $k^*$-stable, and hence F-stable, so $\pi _{V,G}(X)$ is a $\pi _{V/G,F}$-saturated subset of V / G – that is, $\pi _{V/G,F}^{-1}(\pi _{V/G,F}(\pi _{V,G}(X))) = \pi _{V,G}(X)$. Hence, to show the result claimed, we may replace G with $\Gamma$ and show that $\pi _{V,\Gamma }(X)$ is closed in $V/\Gamma$. Now let $S = k[V]^\Gamma \subseteq k[V]^G$. A homogeneous $f \in R$ belongs to S if and only if $\deg (f)$ is divisible by $m'$ (since then the action of F is killed by the degree).

To show that $\pi _{V,\Gamma }(X)$ is closed in $V/\Gamma$, it is enough to show that for every $x \in \overline{\Gamma \cdot X}$ with closed $\Gamma$-orbit, there exists an $x' \in X$ with $\pi _{V,\Gamma }(x') = \pi _{V,\Gamma }(x)$ (cf. Lemma 4.6). If $x \in \overline{\Gamma \cdot X}$ is unstable and $\Gamma \cdot x$ is closed, then x must actually be 0, so $x \in X$ also. Therefore, we may assume that we have $x \in V_{\mathrm{ss},\Gamma } \cap \overline{\Gamma \cdot X}$ such that $\Gamma \cdot x$ is closed (in V). By the discussion before the proposition, we have a morphism $\phi :\mathbb {W}(X)_{\mathrm{ss},K}/K \rightarrow \mathbb {W}(V)_{\mathrm{ss},\Gamma }/\Gamma$ with closed image C, say. Note that since we are assuming $X_{\mathrm{ss},K} = V_{\mathrm{ss},\Gamma }\cap X$, we have $\Gamma \cdot X_{\mathrm{ss},K} = V_{\mathrm{ss},\Gamma }\cap \Gamma \cdot X$, and this set is dense in $V_{\mathrm{ss},\Gamma }\cap \overline{\Gamma \cdot X}$. The composition $V_{\mathrm{ss},\Gamma }{\mathop {\longrightarrow }\limits ^{\xi _V}} {{\mathbb {W}}}(V)_{\mathrm{ss},\Gamma }{\mathop {\longrightarrow }\limits ^{\eta _{V,\Gamma }}} {{\mathbb {W}}}(V)_{\mathrm{ss},\Gamma }/\Gamma$ takes $\Gamma \cdot X_{\mathrm{ss},K}$ into C, and since C is closed that means that the composition in fact takes all of $V_{\mathrm{ss},\Gamma } \cap \overline{\Gamma \cdot X}$ into C. Therefore, we can find z in $\mathbb {W}(X)_{\mathrm{ss},K}/K$ with $\phi (z) = \eta _{V,\Gamma }(\xi _V(x))$. Tracing back through the definitions, we see that $\phi (z) = \eta _{V,\Gamma }(\xi _V(y))$ for some $y \in X_{\mathrm{ss},K}$. It follows that $\eta _{V,\Gamma }(\xi _V(x))=\eta _{V,\Gamma }(\xi _V(y))$; we claim that in fact $\pi _{V,\Gamma }(x) = \pi _{V,\Gamma }(c\cdot y)$ for some $c \in k^*$. Note that suffices to finish the proof, since in particular $y \in X$, so setting $x'=c\cdot y \in X$ gives us what we want.

Both points x and y lie in $V_{\mathrm{ss},\Gamma }$, so there are homogeneous generators $f_i,f_j\in R$ for which $f_i(x)\ne 0$ and $f_j(y) \ne 0$. By definition of m, there are $m_i,m_j\in \mathbb {N}$ such that $f_i^{m_i}$ and $f_j^{m_j}$ both have degree m. Taking a suitable linear combination of $f_i^{m_i}$ and $f_j^{m_j}$, we can therefore find a homogeneous $f \in R$ of degree m for which $f(x)\ne 0\ne f(y)$. Now we can choose $c\in k^*$ such that $f(x) = f(c\cdot y)$.

Let $f' \in S$ be non-constant and homogeneous; as previously observed, $f' \in S$ means that $\deg (f') = rm'$ for some $r \in \mathbb {N}$. Then $(f')^{p^\alpha }$ has degree rm, so $\frac{(f')^{p^\alpha }}{f^r}$ has degree 0 in the localization $R_f$. Further, since x and $c\cdot y$ have the same image in $(\mathbb {W}(V)_{\mathrm{ss},\Gamma }/\Gamma )_f$, we have $\frac{(f')^{p^\alpha }}{f^r}(x) = \frac{(f')^{p^\alpha }}{f^r}(c\cdot y)$. Since $f(x) = f(c\cdot y)\ne 0$, this in turn implies that $(f')^{p^\alpha }(x) = f'^{p^\alpha }(c\cdot y)$, and hence $f'(x) = f'(c\cdot y)$. Since S is generated by homogeneous elements, we see that $\pi _{V,\Gamma }(x) = \pi _{V,\Gamma }(c\cdot y)$, as required. This finishes the proof. $\square$

Proof of Theorem 5.1

We can choose a G-equivariant closed embedding of X in a G-module V. Let $v\in V^H$ such that v is G-unstable. By the argument in the proof of Proposition 4.1, there exists $\lambda \in Y(N_G(H))$ such that $\lim _{a\rightarrow 0} \lambda (a)\cdot v= 0$. This shows that $(V^H)_{\mathrm{ss},N_G(H)}\subseteq V_{\mathrm{ss},G}$. The G-action commutes with the natural $k^*$-action by scalars, and this preserves the subspace $V^H$ also, so Proposition 5.2 implies that the map $\psi _{V,H}:V^H/N_G(H)\rightarrow V/G$ has closed image.

Now $\overline{G\cdot X^H}\cap V^H\subseteq X\cap V^H= X^H$, so $\overline{G\cdot X^H}\cap V^H= X^H$. Let $x\in \overline{G\cdot X^H}$ such that $G\cdot x$ is closed. Then, since $x\in \overline{G\cdot V^H}$ and $\psi _{V,H}$ has closed image, Lemma 4.6 implies $x\in G\cdot V^H$, so we can write $x=g\cdot v$ for some $v\in V^H$. But then $v \in \overline{G\cdot X^H}\cap V^H =X^H$, so $x\in G\cdot X^H$ and we are done by Lemma 4.6. $\square$

Remark 5.3

For the proof of Theorem 5.1, we only need to apply Proposition 5.2 when the $k^*$-action is the standard action by scalars, so the weighted projectivization is the usual projectivization in this case. However, we do need Proposition 5.2 in this more general set-up to complete the proof of Theorem 1.1 in the next section.

Example 5.4

Let G act on $X:= G$ by conjugation and let H be a maximal torus of G. Assume G is connected. Then $X^H= H$. Since the closed orbits in X are precisely the semisimple conjugacy classes [55], the map $\psi _{X,H}:X^H/N_G(H)\rightarrow X/G$ is surjective—in fact, it is well known that $\psi _{X,H}$ is an isomorphism (cf. Sect. 7). Note, however, that although $G\cdot X^H$ is dense in X, not every element of X belongs to $G\cdot H$ (just take $x\in X$ not semisimple).

6 Proof of Theorem 1.1, Part 3: finiteness

We now complete the proof of Theorem 1.1. The implication (ii)$\implies$(i) follows from Theorem 4.4, so it remains to show that if H is G-completely reducible then the morphism $\psi _{X,H}$ is finite. By Lemma 2.3, we can replace X with a larger affine G-variety, hence without loss we can assume that X is a G-module.

Let $G_1$ be the subgroup of G generated by $G^0$ and H. The inclusion of $X^H$ in X gives rise to a morphism $\psi ^1_{X,H}:X^H/N_{G_1}(H)\rightarrow X/G_1$. We have a commutative diagram

(6.1)

where the vertical arrows are the obvious maps. We may identify X / G with the quotient of $X/G^0$ by the finite group $G/G^0$, so the map $X/G^0\rightarrow X/G$ is finite. This map factorizes as $X/G^0\rightarrow X/G_1\rightarrow X/G$, so the map $X/G_1\rightarrow X/G$ is finite by Remark 2.2(ii). Likewise, the map $X^H/N_{G_1}(H)\rightarrow X^H/N_G(H)$ is finite. Hence both of the vertical maps in (6.1) are finite and surjective. Now using Remark 2.2(ii) we see it is enough to show that $\psi ^1_{X,H}$ is finite. So it is enough to prove that $\psi _{X,H}$ is finite under the assumption that $G= G_1$.

Let Y and $U\subseteq Y^H$ be as in Lemma 2.11 and set $V= X\oplus Y$. We have a G-equivariant closed embedding of X in V given by $x\mapsto (x,0)$. Let $W= \overline{G\cdot V^H}$; then $W^H= V^H$. Note that $G\cdot V^H= G^0\cdot V^H$ by our assumption that $G= G_1$, so W is irreducible. By Lemma 2.3 again, it is enough to prove that $\psi _{W,H}$ is finite.

The subset $X^H\times U$ of $X^H\oplus Y^H= V^H= W^H$ is open and dense, and $G_w= H$ for $w\in X^H\times U$. Next we claim that W has good dimension (for the G-action). To see this, let $y_0\in U$ and set $w_0= (0,y_0)$. Then $G\cdot w_0$ is a G-orbit of maximal dimension in W, and $G\cdot w_0$ is closed (as $G\cdot y_0$ is, by Lemma 2.11), so $w_0$ is a stable point of W for the G-action. The claim now follows from Remark 2.6. By a similar argument, $W^H$ has good dimension for the $N_G(H)$-action. Now since the stable points form an open subset, we can conclude that $G\cdot w$ and $N_G(H)\cdot w$ are closed for generic $w\in W^H$, and it follows from Remark 4.5 that generic fibres of $\psi _{W,H}$ are singletons.

Now consider the normalization ${\widetilde{W}}$ of W. Since the normalization map $\nu _W:{\widetilde{W}}\rightarrow W$ is birational, ${\widetilde{W}}$ contains an open dense subvariety ${\widetilde{O}}$ such that the map ${\widetilde{O}}\rightarrow W$ is an isomorphism onto its image O, and O is open in W. We can take ${\widetilde{O}}$ and O to be G-stable, so the latter meets $G\cdot W^H$. Now $G\cdot W^H$ is constructible and dense in W, so it contains a nonempty open subset of W. Hence, by adjusting O and ${\widetilde{O}}$ if necessary, we can assume that $O\subseteq G\cdot W^H$ and O is $N_G(H)$-stable. Since $\nu _W$ is G-equivariant, we get an isomorphism from ${\widetilde{O}}^H$ onto $O^H$. Let C be the closure in ${\widetilde{W}}$ of ${\widetilde{O}}^H$; then $C\subseteq {\widetilde{W}}^H$ and C is $N_G(H)$-stable. Further, since the open subset ${\widetilde{O}}^H$ in C is isomorphic to the open subset $O^H$ in the irreducible set $W^H$, C is irreducible; it follows from Zariski’s Main Theorem that C is isomorphic to $W^H$. Hence $C\cong W^H=V^H$ carries a vector space structure, and we can identify a point $0_C\in C$ corresponding to the zero $0_V$; we have $\nu _W(0_C)= 0_V$ by construction. Furthermore, the action of $k^*$ on W by scalar multiplication lifts to an action of $k^*$ on ${\widetilde{W}}$ which preserves the closed subset C.

We want to apply Proposition 5.2 to deduce that the map $C/N_G(H) \rightarrow {\widetilde{W}}/G$ has closed image (note that we cannot use Theorem 5.1 directly because C might be properly contained in ${\widetilde{W}}^H$). In order to do this, we choose a $(G\times k^*)$-equivariant embedding i of ${\widetilde{W}}$ in a vector space M such that $0_C$ maps to the zero $0_M\in M$. (For instance, choose $f_1,\ldots , f_s\in k[{\widetilde{W}}]$ for some s such that the $f_i$ generate $k[{\widetilde{W}}]$ as a k-algebra and $f_1(0_C)=\cdots = f_s(0_C)= 0$; we can take M to be the dual of N, where N is a $(G\times k^*)$-stable subspace of $k[{\widetilde{W}}]$ containing all the $f_i$.) Replacing M with the subspace spanned by $i({\widetilde{W}})$, we can assume that $i({\widetilde{W}})$ spans M.

Let $\lambda _0:k^*\rightarrow k^*$ be the identity cocharacter of $k^*$. Now $\{0_V\}$ is the unique closed $k^*$-orbit in W, and each element of W is destabilized to $0_V$ by $\lambda _0$. It follows from Lemma 2.8(i) that $\{0_C\}$ is the unique closed $k^*$-orbit in ${\widetilde{W}}$. Let $0_C\ne {\widetilde{w}}\in {\widetilde{W}}$. The Hilbert–Mumford Theorem implies that $\lim _{a\rightarrow 0} \lambda _0(a)\cdot {\widetilde{w}}= 0_C$ or $\lim _{a\rightarrow 0} (-\lambda _0)(a)\cdot {\widetilde{w}}= 0_C$. In particular, $k^*$ does not fix ${\widetilde{w}}$, so $k^*$ does not fix $\nu _W({\widetilde{w}})$, so $\nu _W({\widetilde{w}})\ne 0_V$. Suppose $\lim _{a\rightarrow 0} (-\lambda _0)(a)\cdot {\widetilde{w}}= 0_C$. Then $\lim _{a\rightarrow 0} (-\lambda _0)(a)\cdot \nu _W({\widetilde{w}})= \nu _W(0_C)= 0_V$. But this is impossible because $\lim _{a\rightarrow 0} \lambda _0(a)\cdot \nu _W({\widetilde{w}})= 0_V$ and $\nu _W({\widetilde{w}})\ne 0_V$. We deduce that $\lim _{a\rightarrow 0} \lambda _0(a)\cdot {\widetilde{w}}= 0_C$. Hence, we can conclude that $k^*$ acts on M with positive weights. Further, C is a closed $(N_G(H)\times k^*)$-stable subset of M. By the same argument as in the proof of Proposition 4.1, if $c \in C$ is G-unstable, then since c is H-fixed and H is G-cr, c is also $N_G(H)$-unstable. Hence $C_{\mathrm{ss},N_G(H)} \subseteq M_{\mathrm{ss},G}$. Thus we can now apply Proposition 5.2 to deduce that $C/N_G(H)$ has closed image in M / G. Since $i_G:{\widetilde{W}}/G\rightarrow M/G$ is injective (Lemma 2.3), we deduce that $C/N_G(H)$ has closed image in ${\widetilde{W}}/G$, as we wanted. This allows us to draw the following commutative diagram:

where by abuse of notation we denote the restriction of $\psi _{{\widetilde{W}},H}$ to $C/N_G(H)$ by the same symbol. The leftmost vertical arrow is the isomorphism induced by the isomorphism $C\cong W^H$ above. The other vertical map is finite (Proposition 3.10) and birational (Lemma 2.9; recall that W has good dimension). By Theorem 5.1, $\psi _{W,H}$ has closed image, and we have just argued that $\psi _{{\widetilde{W}},H}\left( C/N_G(H)\right)$ is closed. But $W= \overline{G\cdot W^H}$, so $\psi _{W,H}$ is surjective, and it follows that $\psi _{{\widetilde{W}},H}\left( C/N_G(H)\right) = {\widetilde{W}}/G$. Since $\psi _{W,H}$ is quasi-finite (Theorem 4.4) and has singletons as generic fibres, the same is true of $\psi _{{\widetilde{W}},H}$. As ${\widetilde{W}}/G$ is normal, it follows from Proposition 3.8 that $\psi _{{\widetilde{W}},H}$ is finite and bijective. This implies that $(\nu _W)_G\circ \psi _{{\widetilde{W}},H}$ is finite. Since the leftmost vertical arrow is an isomorphism, we have that $\psi _{W,H}$ is finite, as required. This completes the proof of Theorem 1.1.

7 Separability of $\psi _{X,H}$

We now consider the question of when $\psi _{X,H}$ is an isomorphism, or close to being one. Before we state our result, we need some terminology.

Definition 7.1

Let H be a subgroup of G. We say that H is a principal stabilizer for the G-variety X if there exists a nonempty open subset U of X such that $G_x$ is G-conjugate to H for all $x\in U$. We say that H is a principal connected stabilizer for the G-variety X if H is connected and there exists a nonempty open subset U of X such that $G_x^0$ is G-conjugate to H for all $x\in U$. It is immediate that if G permutes the irreducible components of X transitively then a principal stabilizer (resp., principal connected stabilizer) is unique up to conjugacy, if one exists.

In characteristic 0, principal stabilizers exist under mild hypotheses: for instance, if X is smooth [50, Proposition 5.3] or if X has good dimension [34, Lemma 3.4]. For a counterexample in positive characteristic, see Example 3.2.

Theorem 7.2

Let X be an affine G-variety. Suppose that: (a) H is a principal stabilizer for $X_{\mathrm{cl}}$; (b) H is G-cr; (c) X / G and $X^H/N_G(H)$ are irreducible; and (d) X / G is normal. Then $\psi _{X,H}$ is finite and bijective. In particular, if $\psi _{X,H}$ is separable then it is an isomorphism.

Observe that this result extends a theorem of Luna and Richardson [34, Theorem  4.2] to positive characteristic; note that in characteristic 0, a reductive group H is automatically G-cr, $\psi _{X,H}$ is automatically separable and principal stabilizers exist, as noted above.

Proof

By Theorem 1.1, $\psi _{X,H}$ is finite, so its fibres are finite. To prove the first assertion of the theorem it is enough, therefore, by Proposition 3.8 to show that $\psi _{X,H}$ is surjective and generic fibres of $\psi _{X,H}$ are singletons. By hypothesis, $G\cdot X^H$ contains a nonempty open subset of $X_{\mathrm{cl}}$. The assumption that X / G is irreducible implies that the action of G is transitive on the irreducible components of $X_{\mathrm{cl}}$ so we can conclude that $\pi _{X,G}(X^H)= \pi _{X,G}(G\cdot X^H)$ contains a nonempty open subset of X / G. As X / G is irreducible, $\psi _{X,H}(X^H/N_G(H))= X/G$. If x is a stable point of $X_{\mathrm{cl}}$ and $G_x= H$ then $\psi _{X,H}^{-1}(\psi _{X,H}(\pi _{X^H,N_G(H)}(x)))$ is a singleton, by Remark 4.5. This proves the first assertion as the set of conjugates of such x is open in $X_{\mathrm{cl}}$. If $\psi _{X,H}$ is separable then the second assertion follows from Zariski’s Main Theorem, as X / G is normal. $\square$

Remark 7.3

The assertion of Theorem 7.2 also holds by a similar argument if we replace the hypothesis that H is a principal stabilizer for $X_{\mathrm{cl}}$ with the hypothesis that H is a principal connected stabilizer for $X_{\mathrm{cl}}$.

Next we study the separability condition. To simplify the arguments below, we consider only the case when X has good dimension for the G-action.

Lemma 7.4

Suppose an affine G-variety X has good dimension and hypotheses (a)–(c) of Theorem 7.2 hold. Then $\psi _{X,H}$ is separable if and only if for generic $x\in X^H$, $T_x(G\cdot x)\cap T_xX^H= T_x(N_G(H)\cdot x)$.

Proof

Clearly $T_x(G\cdot x)\cap T_xX^H\supseteq T_x(N_G(H)\cdot x)$, so the content here is in the reverse inclusion. First we claim that $X^H$ has good dimension for the $N_G(H)$-action. To see this, observe that $\overline{G\cdot X^H}= X$ by the surjectivity assertion of Theorem 7.2 (which does not depend on hypothesis (d)), so every closed G-orbit in X meets $X^H$ by Lemma 4.6. As H is a principal stabilizer for X, we must have $G_x= H$ for generic $x\in X^H$, and it follows from Proposition 4.1 that generic $N_G(H)$-orbits in $X^H$ are closed, as required. We now see from Remark 2.6 that

$$\begin{aligned} \pi _{X,G}^{-1}(\pi _{X,G}(x))= G\cdot x\ \text{ and }\ \ \pi _{X^H,N_G(H)}^{-1}(\pi _{X^H,N_G(H)}(x))= N_G(H)\cdot x \end{aligned}$$

(7.5)

for generic $x\in X^H$. Now $\pi _{X,G}$ and $\pi _{X^H,N_G(H)}$ are separable (Lemma 2.7), and it follows from this and from Eq.  (7.5) that for generic $x\in X^H$, $d_x\pi _{X,G}$ is surjective at x with kernel $T_x(G\cdot x)$ and $d_x\pi _{X^H,N_G(H)}$ is surjective at x with kernel $T_x(N_G(H)\cdot x)$.

The map $\psi _{X,H}$ is surjective and finite (by Theorem 1.1), so it is separable if and only its derivative is an isomorphism for generic points in $X^H/N_G(H)$. The result now follows from the argument above. $\square$

Recall that a pair (G, H) of reductive groups with $H\le G$ is called a reductive pair if ${{\mathfrak {h}}}= {{\,\mathrm{Lie}\,}}(H)$ splits off as a direct H-module summand of ${{\mathfrak {g}}}={{\,\mathrm{Lie}\,}}(G)$, where H acts via the adjoint action of G on ${{\mathfrak {g}}}$, and a subgroup $A \le G$ is called separable in G if

$$\begin{aligned} {{\,\mathrm{Lie}\,}}(C_G(A)) = c_{{{\mathfrak {g}}}}(A) := \{X \in {{\mathfrak {g}}}\mid {{\,\mathrm{Ad}\,}}_G(a)(X) = X \text { for all } a \in A\}. \end{aligned}$$

Proposition 7.6

Suppose an affine G-variety X has good dimension and hypotheses (a)–(c) of Theorem 7.2 hold. Suppose one of the following holds:

• (i) there exists $x\in X$ such that $G_x= H$ and there is an étale slice through x for the G-action;

• (ii) H is separable in G, (G, H) is a reductive pair and there exists $x\in X$ such that $G_x= H$ and $G\cdot x$ is separable.

Then $\psi _{X,H}$ is separable.

Proof

By the argument of Theorem 7.2, $\psi _{X,H}$ is dominant. Suppose first that (i) holds. Let $x\in X$ with $G_x= H$ and let S be an étale slice through x for the G-action. By the definition of étale slices and the proof of [1, Proposition 8.6], there exists a G-stable open neigbourhood U of x in X such that $G_y\le G_x$ for all $y\in S\cap U$ and the obvious maps $G\times (S\cap U)\rightarrow X$ and $(S\cap U)/H\rightarrow X/G$ are étale. As H is a principal stabilizer for X, we can assume after replacing U with a smaller open set that $G_y$ is conjugate to H for all $y\in S\cap U$. We have $G_x= H$ by hypothesis, so it follows that $G_y= H$ for all $y\in S\cap U$. As the set of stable points of X is G-stable, open and nonempty and the set of smooth points of X / G is open and nonempty, there is a nonempty G-stable open subset $U_1$ of U such that $G\cdot y$ is closed and $\pi _{X,G}(y)$ is a smooth point of X / G for all $y\in U_1$.

Since $G\cdot (S\cap U_1)$ is open and $S\cap U_1\subseteq X^H$, $G\cdot (S\cap U_1)$ contains a nonempty open subset of $X^H$. Let $y'\in X^H\cap G\cdot (S\cap U_1)$: say, $y'= g\cdot y$ for some $y\in S\cap U_1$, $g\in G$. Then $G_y= H$ and $G_{y'}$ is G-conjugate to H; but $y'\in X^H$, so $G_{y'}= H$. It follows that $g\in N_G(H)$. We deduce that $X^H\cap G\cdot (S\cap U_1)= N_G(H)\cdot (S\cap U_1)$. So $\pi _{X^H,N_G(H)}(S\cap U_1)= \pi _{X^H,N_G(H)}(N_G(H)\cdot (S\cap U_1))$ contains a nonempty open subset of $X^H/N_G(H)$.

So pick $y\in S\cap U_1$ such that $\pi _{X^H,N_G(H)}(y)$ is a smooth point of $X^H/N_G(H)$. The map $(S\cap U)/H\rightarrow X/G$ is étale, so its derivative is an isomorphism everywhere. Hence the derivative of the map $X^H\rightarrow X/G$ induced by $\pi _{X,G}$ is surjective at y. This in turn implies that the derivative of $\psi _{X,H}$ is surjective at $\pi _{X^H,N_G(H)}(y)$. But $\pi _{X^H,N_G(H)}(y)$ and $\pi _{X,G}(y)$ are smooth points by construction, so $\psi _{X,H}$ is separable.

Now suppose that (ii) holds. We argue along the lines of the proof of [48, Theorem  A]. Let ${{\mathfrak {d}}}$ be an H-module complement to ${{\mathfrak {h}}}$ in ${{\mathfrak {g}}}$. Let $X_0= \{x_1\in X\mid G_{x_1}= H \text { and } G\cdot x_1 \text { is closed and separable}\}$. Let $x_1\in X_0$. Then the orbit map $\kappa _{x_1}:G\rightarrow G\cdot x_1$ gives an isomorphism $\phi :G/H\rightarrow G\cdot x_1$. In particular, the derivative $d_1\phi$ at $1\in G$ gives an isomorphism from ${{\mathfrak {g}}}/{{\mathfrak {h}}}$ to the tangent space $T_{x_1}(G\cdot x_1)$, and it is easily checked that $d_1\phi$ is H-equivariant. It follows that $d_1\kappa _{x_1}$ gives an isomorphism of H-modules from ${{\mathfrak {d}}}$ to $T_{x_1}(G\cdot x_1)$. Now let $\beta \in T_{x_1}(G\cdot x_1)\cap T_{x_1}X^H$. Then $\beta$ is fixed by H, so $\beta = d_1\kappa _{x_1}(\alpha )$ for some $\alpha \in {{\mathfrak {d}}}^H$. As H is separable in G, $\alpha \in \mathrm{Lie}(N_G(H))$. Hence $\beta \in T_{x_1}(N_G(H)\cdot x_1)$.

To finish, it is enough by Lemma 7.4 to show that generic elements of $X^H$ belong to $X_0$. As H is a principal stabilizer for X and X has good dimension for the G-action, $G_{x_1}= H$ and $G\cdot x_1$ is closed for generic $x_1\in X^H$. Now

$$\begin{aligned} \mathrm{dim}(G_{x_1})+ \mathrm{dim}(\mathrm{ker}(d_1\kappa _{x_1}))\ge 2\,\mathrm{dim}(H) \end{aligned}$$

(7.7)

for all $x_1\in X^H$. But equality holds in Eq.  (7.7) for $x_1= x$, so it holds for generic $x_1\in X^H$ by Lemma 2.1. This shows that $G\cdot x_1$ is separable for generic $x_1\in X^H$, so we are done. $\square$

The following example shows that separability does not hold automatically under the hypotheses of Theorem 7.2, not even when X has good dimension.

Example 7.8

Let $G= {{\,\mathrm{SL}\,}}_p(k)$, where k has characteristic p and $p> 2$. Let $e_1,\ldots ,e_p$ be the standard basis vectors for the vector space $V:= k^p$ and let $B_0$ be the standard nondegenerate symmetric bilinear form on $k^p$ given by $B_0(e_i,e_j)= \delta _{ij}$. Now let Y be $S^2(V)^*$, the vector space of symmetric bilinear forms on $k^p$; then G acts on Y by $(g\cdot B)(v,w)= B(g^{-1}\cdot v, g^{-1}\cdot w)$. If $B\in Y$ then B is nondegenerate if and only if the $p\times p$ matrix with i, j-entry $B(e_i,e_j)$ has nonzero determinant, so the subvariety X of nondegenerate forms is open and affine. Moreover, X has good dimension since the G-orbits on X all have the same dimension.

The stabilizer $G_{B_0}$ is the special orthogonal group $H:= \mathrm{SO}_p(k)$, and H is G-cr as $\mathrm{char}(k)\ne 2$ (in fact, H is contained in no proper parabolic subgroups of G, so H is “G-irreducible”). It is easily seen that $X^H= \{cB_0\mid c\in k^*\}$ and $N_G(H)= H$; hence $N_G(H)$ acts trivially on $X^H$. Moreover, $X= G\cdot X^H$. Hence H is a principal stabilizer and X has good dimension for the action of G on X.

Let $0\ne B= cB_0\in X^H$. Define $\lambda \in Y(G)$ by $\lambda (a)= \mathrm{diag}(a^{-1},\dots , a^{-1}, a^{p-1})$ (the diagonal matrix with given entries with respect to the basis $e_1,\ldots , e_p$). Let $B_1\in Y$ be the degenerate form given by $B_1(a_1e_1+ \cdots + a_pe_p, b_1e_1+ \cdots + b_pe_p)= ca_pb_p$. Then for all $a\in k^*$, $\lambda (a)\cdot B= a^2B+ (a^{2- 2p}- a^2)B_1$. As X is open in Y, we may identify $T_BX$ with $T_BY$. Making the usual identification of the tangent spaces $T_1k^*$ and $T_BY$ with k and Y, respectively, we see that

$$\begin{aligned} d_1\kappa _B(1)= 2B \end{aligned}$$

(note that since $\mathrm{char}(k)= p$, we have $\displaystyle \left. \frac{\mathrm{d}}{\mathrm{d}a} (a^{{2- 2p}}- a^2)\right| _{a= 1}= 0$). Now $d_1\kappa _B(1)$ belongs to $T_B(G\cdot B)$ and to $T_B(X^H)$, but not to $T_B(N_G(H)\cdot B)$ since the latter tangent space is zero. It follows from Lemma 7.4 that $\psi _{X,H}$ is not separable.

8 Examples

The constructions in Lemma 4.3 demonstrate the failure of Theorem 1.1 when the hypothesis of complete reducibility is removed. In this section we provide some concrete and straightforward examples of this phenomenon.

Example 8.1

Let the characteristic be 2 and let $\rho :{{\,\mathrm{SL}\,}}_2(k) \rightarrow {{\,\mathrm{SL}\,}}_3(k)$ be the adjoint representation of ${{\,\mathrm{SL}\,}}_2(k)$. Concretely, let

$$\begin{aligned} e=\left( \begin{array}{cc} 0 &{} \quad 1\\ 0 &{} \quad 0 \end{array}\right) , h=\left( \begin{array}{cc} 1 &{} \quad 0\\ 0 &{} \quad 1 \end{array}\right) , f=\left( \begin{array}{cc} 0 &{} \quad 0\\ 1 &{} \quad 0 \end{array}\right) \end{aligned}$$

be the standard basis for $X:={{\,\mathrm{Lie}\,}}({{\,\mathrm{SL}\,}}_2(k))$ and let ${{\,\mathrm{SL}\,}}_2(k)$ act on X by conjugation. Then, with respect to this basis, we have

$$\begin{aligned} \rho \left( \begin{array}{cc} a &{} \quad b \\ c &{} \quad d \end{array}\right) = \left( \begin{array}{ccc} a^2 &{} \quad 0 &{} \quad b^2\\ ac &{} \quad 1 &{} \quad bd \\ c^2 &{} \quad 0&{} \quad d^2 \end{array}\right) . \end{aligned}$$

Let H be the image of $\rho$ inside $G = {{\,\mathrm{SL}\,}}_3(k)$ with natural module X. Then H is reductive, but H is not G-cr since the representation $\rho$ is not semisimple: the H-fixed subspace of X spanned by the vector h has no H-stable complement. Since H is reductive, $N_G(H)^0 = H^0C_G(H)^0$. Direct calculation shows that $C_G(H)$ is finite and hence $N_G(H)/H$ is finite. Now the vector h is H-fixed but has a non-closed G-orbit, since if we let $\lambda \in Y(G)$ be the cocharacter defined by

$$\begin{aligned} \lambda (a) := \left( \begin{array}{ccc} 1 &{} \quad 0 &{} \quad 0\\ 0 &{} \quad a &{} \quad 0 \\ 0 \quad &{} 0 \quad &{} \quad a ^{-1}\end{array}\right) \end{aligned}$$

for each $a \in k^*$, then $\lambda (a)\cdot h = ah$, so $\lim _{a\rightarrow 0} \lambda (a) \cdot h = 0$. It is obvious that 0 is not G-conjugate to h. Note that the same reasoning works for any nonzero multiple of h. On the other hand, the $N_G(H)$-orbit of any nonzero multiple of h is finite (and hence closed), and there are therefore infinitely many such closed $N_G(H)$-orbits. Hence the fibre of $\psi _{X,H}$ over $\pi _{X,G}(0)$ is infinite.

Note that this example only works in characteristic 2 because it relies on the existence of the H-fixed vector h. This is consistent with the results above, since away from characteristic 2 the image of the adjoint representation of ${{\,\mathrm{SL}\,}}_2(k)$ in ${{\,\mathrm{SL}\,}}_3(k)$is completely reducible—actually, it is irreducible—and hence is ${{\,\mathrm{SL}\,}}_3(k)$-cr.

Example 8.2

We now provide an infinite family of examples generalizing the previous one. In these examples, G is ${{\,\mathrm{SL}\,}}_m(k)$ acting on its natural module X, and H is the image of some reductive group under a representation in ${{\,\mathrm{SL}\,}}(X)$. Since G has only one closed orbit in X (the orbit $\{0\}$), the quotient X / G is just a single point.

First we consider polynomial representations of ${{\,\mathrm{GL}\,}}_n(k)$ where k is an algebraically closed field of positive characteristic p. A good reference for the polynomial representation theory of ${{\,\mathrm{GL}\,}}_n(k)$ is the monograph [21]. Further details may also be found in the monograph [20]. (To apply this here one should take $q=1$ in the set-up considered there.)

Let the characteristic be $p>0$ and let $G={{\,\mathrm{GL}\,}}_n(k)$ be the group of $n\times n$-invertible matrices. The irreducible polynomial representations of G are parametrized by partitions with at most n parts. More precisely, let $\Lambda ^+(n)$ be the set of partitions $\lambda =(\lambda _1,\dots ,\lambda _n)$ with $\lambda _1\ge \cdots \ge \lambda _n\ge 0$. We may regard $\lambda$ as a weight of the standard maximal torus of G: we set $\lambda (t)= t_1^{\lambda _1}\dots t_n^{\lambda _n}$. Then for each $\lambda \in \Lambda ^+(n)$ there exists an irreducible polynomial G-module $L(\lambda )$ such that $L(\lambda )$ has unique highest weight $\lambda$ and $\lambda$ occurs as a weight with multiplicity one. The modules $L(\lambda ), \lambda \in \Lambda ^+(n)$, form a complete set of pairwise non-isomorphic polynomial irreducible G-modules. We write T for the maximal torus of G consisting of diagonal matrices and B for the subgroup of G consisting of all invertible lower triangular matrices. We shall also need modules induced from B to G. We denote by $k_\lambda$ the 1-dimensional rational T-module on which $t\in T$ acts as multiplication by $\lambda (t)$. The action of T on $k_\lambda$ extends to an action of B. For each $\lambda \in \Lambda ^+(n)$ the induced module $\nabla (\lambda ):= \mathrm{ind}_B^Gk_\lambda$ is a non-zero polynomial representation of G. Then $\nabla (\lambda )$ is finite-dimensional and contains the irreducible module $L(\lambda )$: in fact the G-socle of $\nabla (\lambda )$ is $L(\lambda )$.

We consider the induced ${{\,\mathrm{GL}\,}}_n(k)$-module $\nabla (n(p-1))$. We have that $\nabla (n(p-1))=S^{n(p-1)}E$, where $S^{n(p-1)}E$ is the $n(p-1)$th symmetric power of the natural ${{\,\mathrm{GL}\,}}_n(k)$-module E.

By [19, Lemma  3.3] and [20, 4.3, (10)], the ${{\,\mathrm{GL}\,}}_n(k)$-module $\nabla (n(p-1))$ has simple head $L(p-1,\dots ,p-1)$, which is the 1-dimensional module obtained as the $(p-1)$th tensor power of the determinant module $D=L(1,\dots ,1)$ of ${{\,\mathrm{GL}\,}}_n(k)$. Now let $\Delta (n(p-1))$ be the Weyl module corresponding to the partition $(n(p-1))$. This is the contravariant dual of $\nabla (n(p-1))$. Since $\nabla (n(p-1))$ has simple head we get that $\Delta (n(p-1))$ has simple socle; more precisely,

$$\begin{aligned} {{\,\mathrm{soc}\,}}_{{{\,\mathrm{GL}\,}}_n(k)}(\Delta (n(p-1)))=L(p-1,\dots ,p-1)=D^{\otimes (p-1)}. \end{aligned}$$

Now consider $\Delta (n(p-1))$ as an ${{\,\mathrm{SL}\,}}_n(k)$-module in the usual way. As an ${{\,\mathrm{SL}\,}}_n(k)$-module, $\Delta (n(p-1))$ is the Weyl module corresponding to the dominant weight $(n(p-1))$ and by the considerations above we get that it has simple socle; in particular, ${{\,\mathrm{soc}\,}}_{{{\,\mathrm{SL}\,}}_n(k)}(\Delta (n(p-1)))=L(0)=k$ is the trivial ${{\,\mathrm{SL}\,}}_n(k)$-module. Moreover, since $\Delta (n(p-1))$ is multiplicity-free as an ${{\,\mathrm{SL}\,}}_n(k)$-module we have that L(0) appears as a composition factor of $\Delta (n(p-1))$ with multiplicity 1.

We consider the matrix representation obtained by the ${{\,\mathrm{SL}\,}}_n(k)$-module $\Delta (n(p-1))$. Hence we have a group homomorphism

$$\begin{aligned} \rho :{{\,\mathrm{SL}\,}}_n(k)\rightarrow {{\,\mathrm{SL}\,}}_m(k), \end{aligned}$$

where $m=\dim (\Delta (n(p-1)))=\left( {\begin{array}{c}np-1\\ np-n\end{array}}\right)$. Let $X=\Delta (n(p-1))$ and let H be the image of $\rho$ inside $G= {{\,\mathrm{SL}\,}}_m(k)= {{\,\mathrm{SL}\,}}(X)$. The previous reasoning shows that X is an indecomposable H-module and the trivial module appears in the H-socle of X. The group H is reductive but not G-cr since the representation X is not semisimple.

Since H is reductive we have that $N_G(H)^0=H^0 C_G(H)^0$. Moreover, ${{\,\mathrm{End}\,}}_H(X)={{\,\mathrm{End}\,}}_{{{\,\mathrm{SL}\,}}_n(k)}(X)=k$ (see [26, Proposition 2.8]); this implies that $C_G(H)$ is finite, so $N_G(H)/H$ is finite.

Now the quotient $X^H/N_G(H)$ is infinite since H fixes a full 1-dimensional subspace of X and $N_G(H)/H$ is finite. On the other hand, the quotient X / G is a single point and so the morphism

$$\begin{aligned} \psi _{X,H}:X^H/N_G(H)\rightarrow X/G \end{aligned}$$

is not a finite morphism.

Note that Example 8.1 above is just this one with $p=n=2$.

Example 8.3

We provide another example, this time with a symplectic group. Let $p=2$ and consider the symplectic group ${{\,\mathrm{Sp}\,}}_4(k)$. We choose the simple roots $\alpha =(2,-1)$ and $\beta =(-2,2)$. The simple ${{\,\mathrm{Sp}\,}}_4(k)$-module L(0, 1), corresponding to the dominant weight (0, 1), is 4-dimensional with weights $(0,1), (2,-1), (-2,1),(0,-1)$. We consider the Weyl module $\Delta (0,1)$ corresponding to (0, 1). This is an indecomposable 5-dimensional module with simple head L(0, 1) and it fits into the short exact sequence

$$\begin{aligned} 0\rightarrow k\rightarrow \Delta (0,1)\rightarrow L(0,1)\rightarrow 0, \end{aligned}$$

where k is the trivial ${{\,\mathrm{Sp}\,}}_4(k)$-module.

Now consider the matrix representation corresponding to the ${{\,\mathrm{Sp}\,}}_4(k)$-module $\Delta (0,1)$. This gives a group homomorphism

$$\begin{aligned} \rho :{{\,\mathrm{Sp}\,}}_4(k)\rightarrow {{\,\mathrm{SL}\,}}_5(k). \end{aligned}$$

Let $X=\Delta (0,1)$ and let H be the image of ${{\,\mathrm{Sp}\,}}_4(k)$ in $G={{\,\mathrm{SL}\,}}_5(k)$ with natural module X. Then X is an indecomposable H-module and the trivial module appears in the H-socle of X. The group H is reductive but not G-cr since the representation X is not semisimple.

Since H is reductive we have that $N_G(H)^0=H^0 C_G(H)^0$. Moreover, we have that ${{\,\mathrm{End}\,}}_H(X)={{\,\mathrm{End}\,}}_{{{\,\mathrm{Sp}\,}}_4(k)}(X)=k$ (see [26, Proposition 2.8]), so the only endomorphisms of X as an H-module are the scalars. Since $G = {{\,\mathrm{SL}\,}}_5(k)$, this means that $C_G(H)$ is finite and so $N_G(H)/H$ is finite. Now, as in our previous examples, the quotient $X^H/N_G(H)$ is infinite since H fixes a full one-dimensional subspace of X and $N_G(H)/H$ is finite, whereas the quotient X / G is a single point. Therefore the morphism $\psi _{X,H}:X^H/N_G(H)\rightarrow X/G$ is not a finite morphism.

Example 8.4

The above examples show that if H is the image of a non-completely reducible representation of a reductive group in $G= {{\,\mathrm{GL}\,}}(X)$ or ${{\,\mathrm{SL}\,}}(X)$ then the conclusion of Theorem 1.1 can fail. On the other hand, if H is the image of a completely reducible representation then we get an easy representation-theoretic proof of Theorem 1.1 in this special case, as follows. If the representation is trivial (of any dimension), so that H is the trivial group, then $X^H = X$ and $N_G(H) = G$, so the map $\psi _{X,H}$ is the identity map. If the representation is non-trivial and irreducible, then $X^H = \{0\}$ and the map $\psi _{X,H}:X^H/N_G(H)\rightarrow X/G$ is just the map from a singleton set to a singleton set and hence is finite. If the representation is non-trivial and completely reducible but not irreducible then $X^H$ has an H-complement in X: say, $X = X^H \oplus W$. The centre of the Levi subgroup of G corresponding to the given decomposition normalizes H and acts as scalars on $X^H$, so $X^H/N_G(H)$ is again a singleton set and $\psi _{X,H}$ is finite.

9 Double cosets

In this section we consider a separate but related problem, using techniques from earlier sections. Fix a reductive group G, and reductive subgroups H and K of G. The group $H\times K$ acts on G by the formula $(h,k)\cdot g= hgk^{-1}$; the orbits of the action are the (H, K)-double cosets and we call this action the double coset action. The stabilizer $(H\times K)_g$ is given by $\{(h,g^{-1}hg)\mid h\in H\cap gKg^{-1}\}$. We are interested in the following question: when does G have good dimension for the double coset action? Note that, again, in characteristic 0 this problem was solved by Luna in [31]; he showed using étale slices that G always has good dimension for the double coset action. The problem of translating Luna’s results to positive characteristic was also studied by Brundan [12,13,14, 16], who considered in particular the question of when there is a dense double coset in G. Our main result gives a necessary and sufficient condition for G to have good dimension for the double coset action in terms of the stabilizers of the action.

Theorem 9.1

Let G be connected. The following are equivalent:

• (i) G has good dimension for the $(H\times K)$-action;

• (ii) generic stabilizers of $H\times K$ on G are reductive;

• (iii) $H\cap gKg^{-1}$ is reductive for generic $g\in G$.

Remarks 9.2

(i). It follows from [37, Theorem  1.1] that in order to show that generic stabilizers are reductive, it is enough to show that $(H\times K)_g$ has minimal dimension and is reductive for some $g\in G$.

(ii). Work of Popov [45] implies that if a connected semisimple group G acts on a smooth irreducible affine variety V and the divisor class group $\mathrm{Cl}(V)$ has no elements of infinite order then generic orbits of G on V are closed if and only if generic stabilizers of G on V are reductive. By work of Tange [59, Theorem  1.1], if G is connected then $\mathrm{Cl}(G)$ has no elements of infinite order, so Theorem 9.1 follows if H and K are connected and semisimple.

We need some preparatory results and notation. First, given a cocharacter $\tau = (\lambda ,\mu )\in Y(H\times K)$ and $g\in G$, we say that $\tau$destabilizes g if $\lim _{a\rightarrow 0} \tau (a)\cdot g = \lim _{a\rightarrow 0} \lambda (a)g\mu (a)^{-1}$ exists. Given $g\in G$, define a homomorphism ${\widehat{\phi }}_g:G\rightarrow G\times G$ by ${\widehat{\phi }}_g(g')= (g',g^{-1}g'g)$. A short calculation shows that ${\widehat{\phi }}_g$ induces an isomorphism $\phi _g:H\cap gKg^{-1}\rightarrow (H\times K)_g$. This shows that (ii) and (iii) of Theorem 9.1 are equivalent. Moreover, given $g\in G$ we define an isomorphism of varieties $r_g:G\rightarrow G$ by $r_g(g')= g'g^{-1}$ and an isomorphism of algebraic groups $\psi _g:H\times K\rightarrow H\times gKg^{-1}$ by $\psi _g(h,k)= (h,gkg^{-1})$; then $r_g$ is a $\psi _g$-equivariant map from the $(H\times K)$-variety G to the $(H\times gKg^{-1})$-variety G, where we let $H\times gKg^{-1}$ act on G by the double coset action.

Lemma 9.3

Let $g\in G$. Then G has good dimension for the $(H\times K)$-action if and only if G has good dimension for the $H\times gKg^{-1}$-action, and generic stabilizers of $H\times K$ on G are reductive if and only if generic stabilizers of $H\times gKg^{-1}$ on G are reductive.

Proof

The $\psi _g$-equivariance of $r_g$ implies that $(H\times gKg^{-1})\cdot r_g(g')= r_g((H\times K)\cdot g')$ and $r_g((H\times K)_{g'})= (H\times gKg^{-1})_{r_g(g')}$ for all $g'\in G$. The result follows. $\square$

In the special case when A is reductive, the next result is [35, Lemma  4.1]. We take the opportunity to correct the proof given in loc. cit.

Lemma 9.4

Let $k'$ be an algebraically closed extension field of k. Let A be a linear algebraic group acting on an affine variety X, and let $A'$ (resp. $X'$) be the group (resp. variety) over $k'$ obtained from K (resp. X) by extension of scalars. Let $x\in X$. Then:

• (i) $\mathrm{dim}_{k'}(A'_x)= \mathrm{dim}_k(A_x)$ and ${\dim }_{k'}(A'\cdot x)= \mathrm{dim}_k(A\cdot x)$;

• (ii) $A'\cdot x$ is closed in $X'$ if and only if $A\cdot x$ is closed in X.

Proof

We regard X as a subset of $X'$ and A as a subgroup of $A'$ in the obvious way. The orbit map $\kappa _x:A'\rightarrow A'\cdot x$ is defined over k, so the closure $\overline{A'\cdot x}$ (in $X'$) is k-defined [10, Cors. AG.14.5 and AG.14.6]. This implies that $\overline{A'\cdot x}\cap X= \overline{A\cdot x}$, where the RHS is the closure in X. The stabilizer $A'_x$ is k-defined—in fact, $A'_x$ is naturally isomorphic to the group over $k'$ obtained from $A_x$ by extension of scalars. Hence $\mathrm{dim}_{k'}(A'_x)= \mathrm{dim}_k(A_x)$. This proves the first assertion of (i), and the second follows immediately.

Let $r= {\dim }_{k'}(A'\cdot x)= \mathrm{dim}_k(A\cdot x)$. Set $X'_t= \{y'\in X'\mid \mathrm{dim}_{k'}(A'_{y'})\ge t\}$ and $X_t= \{y\in X\mid \mathrm{dim}_k(A_y)\ge t\}$ for $t\ge 0$. Then $X'_t$ and $X_t$ are closed in $X'$ and X, respectively, and it follows from the proof of [44, Lemma 3.7(c)] that $X'_t$ is k-defined. By (i), $X_t= X'_t\cap X$. Now $\overline{A'\cdot x}$ is the union of $A'\cdot x$ with certain other $A'$-orbits, each of which has dimension strictly less than r, and likewise for $\overline{A\cdot x}$. Hence $A'\cdot x$ (resp., $A\cdot x$) is closed if and only if $\overline{A'\cdot x}\cap X'_{r+1}= \emptyset$ (resp., $\overline{A\cdot x}\cap X_{r+1}= \emptyset$). But $\overline{A'\cdot x}\cap X'_{r+1}$ is k-defined and k is algebraically closed, so $\overline{A'\cdot x}\cap X'_{r+1}$ is empty if and only if $(\overline{A'\cdot x}\cap X'_{r+1})\cap X= \overline{A\cdot x}\cap X_{r+1}$ is empty. Part (ii) now follows. $\square$

Lemma 9.5

Assume G is connected. Let $k'$ be an algebraically closed extension field of k and let $G'$, $H'$ and $K'$ be the algebraic groups over $k'$ obtained from G, H and K, respectively, by extension of scalars. Then:

• (i) generic stabilizers of $H'\times K'$ on $G'$ are reductive if and only if generic stabilizers of $H\times K$ are reductive;

• (ii) $G'$ has good dimension for the $(H'\times K')$-action if and only if G has good dimension for the $(H\times K)$-action.

Proof

We can regard G, H and K as dense subgroups of $G'$, $H'$ and $K'$, respectively. If $g\in G$ then $(H'\times K')_g$ is isomorphic to the group obtained from $(H\times K)_g$ by extension of scalars, so $(H'\times K')_g$ is reductive if and only if $(H\times K)_g$ is reductive. By Lemma 9.4, $(H'\times K')\cdot g$ is closed in $G'$ if and only if $(H\times K)\cdot g$ is closed in G, and $\mathrm{dim}((H'\times K')_g)$ is minimal if and only if $\mathrm{dim}((H\times K)_g)$ is minimal, so g is a stable point for the $(H'\times K')$-action if and only if it is a stable point for the $(H\times K)$-action. The union of the stable $(H\times K)$-orbits is open in G, and likewise for $H'\times K'$ and $G'$ (Lemma 2.6). The union of the $(H\times K)$-orbits of minimum dimension having reductive stabilizer is open in G, and likewise for $H'\times K'$ and $G'$ [37, Theorem  1.1]. Putting these facts together, we obtain the desired result. $\square$

Lemma 9.6

Let $\lambda \in Y(H)$, $\mu \in Y(K)$. Given $g\in G$ such that $(\lambda ,\mu )$ destabilizes g, set $g_0:= \lim _{a\rightarrow 0} \lambda (a)g\mu (a)^{-1}$ and let $u=g g_0^{-1}$. Then $\mu = g_0^{-1}\cdot \lambda$ and $u\in R_u(P_\lambda )$.

Proof

Since $g_0$ is obtained as a limit along $(\lambda ,\mu )$, we have that $(\lambda ,\mu )$ fixes $g_0$, so $\lambda (a)g_0\mu (a)^{-1} = g_0$ for all $a\in k^*$. Rearranging, we see that $\mu = g_0^{-1}\cdot \lambda$. Now for all $a \in k^*$,

$$\begin{aligned} \lambda (a)g\mu (a)^{-1} = \lambda (a)ug_0\mu (a)^{-1} = \lambda (a)ug_0(g_0^{-1} \lambda (a)^{-1}g_0) = \lambda (a)u\lambda (a)^{-1}g_0. \end{aligned}$$

As $\lim _{a\rightarrow 0} \lambda (a)g\mu (a)^{-1}= g_0$, it follows that $\lim _{a\rightarrow 0} \lambda (a)u\lambda (a)^{-1}= 1$, so $u\in R_u(P_\lambda )$. $\square$

Lemma 9.7

Assume G is connected. Let $G_1= G/Z(G)^0$, let $\sigma :G\rightarrow G_1$ be the canonical projection and set $H_1= \sigma (H)$ and $K_1= \sigma (K)$. Then for all $g\in G$:

• (i) $(H\times K)_g$ is reductive if and only if $(H_1\times K_1)_{\sigma (g)}$ is reductive;

• (ii) if $(H_1\times K_1)\cdot \sigma (g)$ is closed then $(H\times K)\cdot g$ is closed.

Proof

(i) Let ${\widetilde{H}}= HZ(G)^0$ and let ${\widetilde{K}}= KZ(G)^0$. Let $A= (\sigma \times \sigma )^{-1}((H_1\times K_1)_{\sigma (g)})$, a subgroup of ${\widetilde{H}}\times {\widetilde{K}}$. Define $\psi :A\rightarrow G$ by $\psi ({\widetilde{h}}, {\widetilde{k}})= g^{-1}{\widetilde{h}}g{\widetilde{k}}^{-1}$. A short calculation shows that $\psi$ gives a homomorphism from A to $Z(G)^0$, with kernel $(H\times K)_g$. Moreover, $\sigma \times \sigma$ gives an epimorphism from A to $(H_1\times K_1)_{\sigma (g)}$, with kernel $Z(G)^0\times Z(G)^0$. Part (i) now follows.

(ii) Let $g\in G$ and suppose $(H_1\times K_1)\cdot \sigma (g)$ is closed. Let $(\lambda ,\mu )\in Y(H\times K)$ such that $g':= \lim _{a\rightarrow 0} \lambda (a)g\mu (a)^{-1}$ exists. Set $g_1= \sigma (g)$, $g_1'= \sigma (g')$. Let $\lambda _1= \sigma \circ \lambda \in Y(H_1)$ and $\mu _1 = \sigma \circ \mu \in Y(K_1)$; then $g_1'= \lim _{a\rightarrow 0} \lambda _1(a)g_1\mu _1(a)^{-1}$. By hypothesis, $g_1'$ is $(H_1\times K_1)$-conjugate to $g_1$. Now the group $Z(G)^0$ acts on G by right inverse multiplication, and we can identify $\sigma$ with the canonical projection to the quotient. The orbits of $Z(G)^0$ all have the same dimension, so $\sigma$ is a geometric quotient. Moreover, the $Z(G)^0$-action commutes with the $(H\times K)$-action, so $H\times K$ acts on $G_1$. By construction, $(h,k)\cdot \sigma (x)= (\sigma (h),\sigma (k))\cdot \sigma (x)$ for all $x\in G$, $h\in H$ and $k\in K$. In particular, $g_1'$ is $(H\times K)$-conjugate to $g_1$. It follows from [9, Cor. 3.5(ii)] that $g'$ is $(H\times K)$-conjugate to g. Hence $(H\times K)\cdot g$ is closed. This proves (ii). $\square$

Lemma 9.8

Suppose G, H and K are connected. Let $\lambda \in Y(H)$. Suppose there exists a nonempty subset C of G such that $(H\times K)\cdot C$ is open and has the following property: for all $g\in C$, there exists $\tau _g= (\lambda ,\mu _g)\in Y(H\times K)$ such that $\tau _g$ destabilizes g.

• (i) There exists $g_0\in G$ such that $\lambda \in Y(g_0Kg_0^{-1})$ and $(H\times g_0Kg_0^{-1})\cdot P_\lambda$ is dense in G. Moreover, for all $g\in P_\lambda$, the cocharacter $(\lambda ,\lambda )$ of $H\times g_0Kg_0^{-1}$ destabilizes g.

• (ii) Suppose in addition that $\tau _g$ fixes g for all $g\in C$. Then $(H\times g_0Kg_0^{-1})\cdot L_\lambda$ is dense in G, and $(\lambda ,\lambda )$ fixes every $l\in L_\lambda$.

Proof

Fix $v\in C$ and let $v_0:= \lim _{a\rightarrow 0} \tau _v(a)\cdot v= \lim _{a\rightarrow 0} \lambda (a)v\mu _v(a)$. Then $\lambda = v_0\cdot \mu _v$ by Lemma 9.6, so $\lambda \in Y(v_0Kv_0^{-1})$. The equivariance of $r_{v_0}$ implies that for any $w\in C$, $(\lambda , v_0\cdot \mu _w)\in Y(H\times v_0Kv_0^{-1})$ destabilizes $wv_0^{-1}$ to $w_0v_0^{-1}$, where $w_0:= \lim _{a\rightarrow 0} \tau _w(a)\cdot w$. By Lemma 9.3, we can replace K with $v_0Kv_0^{-1}$ and C with $Cv_0^{-1}$. So without loss we assume that $\lambda \in Y(K)$.

Let $g\in C$. By hypothesis, $\tau _g = (\lambda ,\mu _g)$ destabilizes g. As $\mathrm{im}(\lambda )$ is contained in K, there exists $k\in K$ such that $\mu := k\cdot \mu _g$ commutes with $\lambda$. Set $g_1= gk^{-1} = (1,k)\cdot g$, so that $(\lambda ,\mu )$ destabilizes $g_1$. Finally, set $g_2= \lim _{a\rightarrow 0} \lambda (a)g_1\mu (a)^{-1}$. Then $\lambda = g_2\cdot \mu$ by Lemma 9.6. Fix a maximal torus T of G such that $\lambda ,\mu \in Y(T)$ and let $n_1,\ldots , n_r\in N_G(T)$ be a set of representatives for the Weyl group $N_G(T)/T$. Now $g_2Tg_2^{-1}$ is a maximal torus of $L_\lambda$, so by conjugacy of maximal tori in $L_\lambda$, we have $xg_2Tg_2^{-1}x^{-1}= T$ for some $x\in L_\lambda$. Then $xg_2= tn_i$ for some i and some $t \in T$, so $g_2= ln_i$, where $l:= x^{-1}t\in L_\lambda$. By Lemma 9.6, we have $g_1= ug_2= uln_i$ for some $u\in R_u(P_\lambda )$, so $g_1\in P_\lambda n_i$ and $g = g_1k \in (H\times K)\cdot (P_\lambda n_i)$. Since $g\in C$ was arbitrary, it now follows that $\bigcup _{i=1}^r (H\times K)\cdot (P_\lambda n_i)$ contains $(H\times K)\cdot C$ and, since G is connected, $(H\times K)\cdot (P_\lambda n_i)$ is dense in G for at least one i. Note also that $\lambda = g_2\cdot \mu = ln_i\cdot \mu$, so $\mu = n_i^{-1}l^{-1}\cdot \lambda = n_i^{-1}\cdot \lambda$, so $\lambda = n_i\cdot \mu \in Y(n_iKn_i^{-1})$.

Keeping the notation in the previous paragraph, for each i, $(H\times K)\cdot (P_\lambda n_i)$ is constructible, so $(H\times K)\cdot (P_\lambda n_i)$ is either dense or contained in a proper closed subset of G. Thus the union of those subsets $(H\times K)\cdot (P_\lambda n_i)$ that are dense contains an open subset of G; note also that this union is $(H\times K)$-stable. Since $(H\times K)\cdot C$ is open, we can find $g'\in C$ such that for any i, if $g'\in (H\times K)\cdot (P_\lambda n_i)$ then $(H\times K)\cdot (P_\lambda n_i)$ is dense. By the arguments in the paragraph above applied to $g'$, there exists i such that $g'\in (H\times K)\cdot (P_\lambda n_i)$ and for this i we have $\lambda = n_i\cdot \mu \in Y(n_iKn_i^{-1})$; moreover, $(H\times K)\cdot (P_\lambda n_i)$ is dense by construction. It follows that $(H\times n_iKn_i^{-1})\cdot P_\lambda = r_{n_i}((H\times K)\cdot (P_\lambda n_i))$ is dense in G, so the first assertion of part (i) follows with $g_0 = n_i$. It is obvious that $(\lambda ,\lambda )$ destabilizes g for all $g \in P_\lambda$, so we have proved part (i).

If $g\in C$ and $\tau _g$ fixes g then $(\lambda ,\mu )$ fixes $g_1$, so $g_1= g_2\in L_\lambda n_i$ for some i. The first assertion of (ii) follows by a similar argument to that above but applied to $\bigcup _{i=1}^r (H\times K)\cdot (L_\lambda n_i)$, and the second assertion is again obvious. $\square$

Proof of Theorem 9.1

We have shown already that (ii) and (iii) are equivalent, so it is enough to prove that (i) and (ii) are equivalent. First note that for any $g\in G$, $(H\cap K)\cdot g$ is closed if and only if $(H\cap K)^0\cdot g= (H^0\cap K^0)^0\cdot g$ is closed, and $H\cap gKg^{-1}$ is reductive if and only if $(H\cap gKg^{-1})^0= (H^0\cap gK^0g^{-1})^0$ is reductive, which is the case if and only if $H^0\cap gK^0g^{-1}$ is reductive. Hence we can assume that H and K are connected. Moreover, we can assume by Lemma 9.5 that k is uncountable.

The implication (i)$\implies$(ii) follows immediately from Lemma 3.3(ii). For the reverse implication, we use induction on $\mathrm{dim}(G)$. Suppose generic stabilizers are reductive. The result is immediate if $\mathrm{dim}(G)= 0$. If G is not semisimple then let $G_1$, $\sigma$, $H_1$ and $K_1$ be as in Lemma 9.7. Then generic stabilizers of $H_1\times K_1$ on $G_1$ are reductive, by Lemma 9.7(i). Since $\mathrm{dim}(G_1)< \mathrm{dim}(G)$, it follows by induction that generic orbits of $H_1\times K_1$ on $G_1$ are closed. Part (ii) of Lemma 9.7 now implies that generic orbits of $H\times K$ on G are closed, so we are done. Hence we can assume that G is semisimple.

First we consider the case when generic stabilizers of $H\times K$ on G are positive-dimensional. Then all stabilizers of $H\times K$ on G are positive-dimensional, by semi-continuity of stabilizer dimension. For each $g\in G$ such that $(H\times K)_g$ is reductive, choose a nontrivial cocharacter $\tau _g\in Y((H\times K)_g)$. The fixed point set $G^{\tau _g}:= G^{\mathrm{im}(\tau _g)}$ is closed, so $C_g:= (H\times K)\cdot G^{\tau _g}$ is constructible. Since generic stabilizers of $H\times K$ on G are reductive, the constructible sets $C_g$ for $g\in G$ such that $(H\times K)_g$ is reductive cover an open dense subset U of G, by [37, Theorem 1.1]. There are only countably many of these sets, as $H\times K$ has only countably many conjugacy classes of cocharacters. By [37, Cor. 2.5], $C_{{\widetilde{g}}}$ is dense in G for some ${\widetilde{g}}\in G$. Hence there exists $\tau = (\lambda ,\mu )\in Y(H\times K)$ such that for generic $g\in G$, g is fixed by an $(H\times K)$-conjugate of $\tau$. It follows from Lemma 9.8 that for some $g_0\in G$, $\lambda \in Y(g_0Kg_0^{-1})$ and $(H\times g_0Kg_0^{-1})\cdot L_\lambda$ is dense in G. By Lemma 9.3, there is no harm in assuming that $g_0Kg_0^{-1}= K$—i.e., that $\lambda \in Y(K)$ and $(H\times K)\cdot L_\lambda$ is dense in G—and we shall do this for notational convenience.

To prove that generic $(H\times K)$-orbits on G are closed, it is therefore enough to show that $(H\times K)\cdot l$ is closed for generic $l\in L_\lambda$. Let $H_2= L_\lambda (H)$ and let $K_2= L_\lambda (K)$; then $H_2\times K_2= L_{(\lambda ,\lambda )}(H\times K)$. Consider the double coset action of $H_2\times K_2$ on $L_\lambda$. Let $l\in L_\lambda$. Then $(\lambda ,\lambda )$ fixes l, so $(H_2\times K_2)_l= L_{(\lambda ,\lambda )}((H\times K)_l)$, which is reductive if $(H\times K)_l$ is. Hence generic stabilizers of $(H_2\times K_2)$ on $L_\lambda$ are reductive. As G is semisimple, $\mathrm{dim}(L_\lambda )< \mathrm{dim}(G)$, so generic $(H_2\times K_2)$-orbits on $L_\lambda$ are closed by induction. It follows from Remark 4.2(iii) that $(H\times K)\cdot l$ is closed for generic $l\in L_\lambda$, so we are done as $(H\times K)\cdot L_\lambda$ is dense in G.

Now consider the case when generic stabilizers of $H\times K$ on G are finite. Suppose generic $(H\times K)$-orbits on G are not closed. Then $G_{\mathrm{cl}}$ is a proper closed subset of G, so the union of the non-closed orbits contains a nonempty open subset of G. For each $g\in G$ such that $(H\times K)\cdot g$ is not closed, choose nontrivial $\tau _g\in Y(H\times K)$ such that $\tau _g$ destabilizes g. By an argument similar to the one in the positive-dimensional case above, there exist $\lambda \in Y(H)$ and $g_0\in G$ such that $\lambda \in Y(g_0Kg_0^{-1})$ and $(H\times g_0Kg_0^{-1})\cdot P_\lambda$ is dense in G. As before, we can assume that $g_0Kg_0^{-1}= K$. Now $R_u(P_{-\lambda })(H)P_\lambda (H)$ and $P_\lambda (K) R_u(P_{-\lambda })(K)$ are nonempty open subsets of H and K respectively [10, Proposition 14.21(iii)], so $R_u(P_{-\lambda })(H)P_\lambda R_u(P_{-\lambda })(K)$ is dense in G, as $HP_\lambda K$ is. It follows that $\mathrm{dim}(R_u(P_{-\lambda })(H))+ \mathrm{dim}(R_u(P_{-\lambda })(K))+ \mathrm{dim}(P_\lambda )\ge \mathrm{dim}(G)$, so $\mathrm{dim}(R_u(P_{-\lambda })(H))+ \mathrm{dim}(R_u(P_{-\lambda })(K))\ge \mathrm{dim}(G)- \mathrm{dim}(P_\lambda )= \mathrm{dim}(R_u(P_\lambda ))$.

By hypothesis, we can choose $g\in P_\lambda$ such that $(H\times K)_g$ is finite. Write $g= ul$, where $l= L_\lambda$ and $u\in R_u(P_\lambda )$; then $l= \lim _{a\rightarrow 0} (\lambda , \lambda )(a)\cdot g$. We show that l is $(H\times K)$-conjugate to g. Consider the double coset action of $R_u(P_\lambda (H))\times R_u(P_\lambda (K))$ on G. Let $O= (R_u(P_\lambda (H))\times R_u(P_\lambda (K)))\cdot g$ and consider the orbit map $\kappa _g:R_u(P_\lambda (H))\times R_u(P_\lambda (K))\rightarrow O$ given by $\kappa _g(h,k)= hgk^{-1}$. It is clear that $O\subseteq R_u(P_\lambda )l$. Note that O is closed, since O is the orbit of an action of a unipotent group on an affine variety [10, Proposition 4.10]. The stabilizer of g in $R_u(P_\lambda (H))\times R_u(P_\lambda (K))$ is finite, since $(H\times K)_g$ is finite, so O has dimension $\mathrm{dim}(R_u(P_\lambda (H)))+ \mathrm{dim}(R_u(P_\lambda (K)))$. Now $\mathrm{dim}(R_u(P_\lambda (H)))+ \mathrm{dim}(R_u(P_\lambda (K)))= \mathrm{dim}(R_u(P_{-\lambda })(H))+ \mathrm{dim}(R_u(P_{-\lambda })(K))\ge \mathrm{dim}(R_u(P_\lambda ))$, and since O is closed, this forces O to be the whole of $R_u(P_\lambda )l$. Hence there exists $(h,k)\in R_u(P_\lambda (H))\times R_u(P_\lambda (K))$ such that $(h,k)\cdot g= l$, as required.

Now $(H\times K)_l$ is finite, since l is $(H\times K)$-conjugate to g. But $(\lambda ,\lambda )$ fixes l, a contradiction. We deduce that generic $(H\times K)$-orbits on G are closed after all. This completes the proof. $\square$

Remark 9.9

One can prove the following more general statement of Theorem 9.1 for non-connected reductive G. Let $G_1,\ldots , G_r$ be the minimal subsets of G having the property that each $G_i$ is $(H\times K)$-stable and contains some connected component of G. Each $G_i$ is a union of certain connected components of G; if H and K are connected then the $G_i$ are precisely the connected components of G. Here is our result: for each i, $G_i$ has good dimension for the $(H\times K)$-action if and only if generic stabilizers of $H\times K$ on $G_i$ are reductive if and only if $H\cap gKg^{-1}$ is reductive for generic $g\in G_i$. To see this, note first that we can assume that H and K are connected, by the proof of Theorem 9.1; hence we can assume that each $G_i$ is a connected component of G. We can now choose $g\in G$ such that $G_ig= G^0$, and use the map $r_g$ to translate the case of $G_i$ into the case of the connected group $G^0$ (cf. the proof of Lemma 9.3). We leave the details to the reader.

We record a useful corollary.

Corollary 9.10

Suppose one of H and K is a torus. Then G has good dimension for the $(H\times K)$-action.

Proof

This is immediate from Theorem 9.1, since any subgroup of a torus is reductive. $\square$

We now consider a concrete example; our methods allow us to deal with arbitrary characteristic. Note that we use Theorem 1.1 in parts (a) and (b) below.

Example 9.11

Let G be simple of type $B_2$ and fix a maximal torus T of G. Let A be the subgroup of G generated by the long root groups with respect to T. If $p= 2$ then let B be the subgroup of G generated by the short root groups with respect to T. The groups A and B are normalized by $N_G(T)$.

(a). Let p be arbitrary and let $H= K= A$. Since $\mathrm{dim}(G)= 10$ and $\mathrm{dim}(H)= \mathrm{dim}(K)= 6$, $\mathrm{dim}(H\times K)_g\ge 2$ for all $g\in G$, with equality if and only if $(H\times K)\cdot g$ is dense in G. Let $\lambda \in Y(T)$ be nontrivial. We show first that for generic $l\in L_\lambda$, $(H\times K)\cdot l$ is closed. If $L_\lambda = T$ or $L_\lambda$ is a long-root Levi subgroup (that is, a Levi subgroup L such that [L, L] is the subgroup of type $A_1$ corresponding to some long root) then $L_\lambda \le A$, so $(H\times K)\cdot l= A$ is closed. Note that in this case, $(H\times K)\cdot L_\lambda = A$ is not dense in G.

So suppose $L_\lambda$ is a short-root Levi subgroup. As in the positive-dimensional case in the proof of Theorem 9.1, it is enough to show that $(L_\lambda (H)\times L_\lambda (K))\cdot l$ is closed for generic $l\in L_\lambda$. But this follows from Corollary 9.10, since $L_\lambda (H)\times L_\lambda (K)= T\times T$ is a torus. Moreover, in this case the quotient space $L_\lambda /(T\times T)$ is positive-dimensional, as $\mathrm{dim}(T\times T)= 4= \mathrm{dim}(L_\lambda )$ and $(T\times T)_l$ has dimension at least 1 for all $l\in L$ (since $(\lambda ,\lambda )$ fixes l). It follows that the quotient space $G/(H\times K)$ is positive-dimensional. To see this, let S be the image of $(\lambda , \lambda )$; note that $T\times T\le N_{H\times K}(S)\le N_{H\times K}(T\times T)$. Now consider the maps $L_\lambda /(T\times T)\rightarrow L_\lambda /N_{H\times K}(S)\rightarrow G/(H\times K)$. The first map is finite as $T\times T$ has finite index in $N_{H\times K}(S)$, while the second is finite by Theorem 1.1 (applied to the subgroup S of $H\times K$), so $\mathrm{dim}(G/(H\times K))\ge \mathrm{dim}(L_\lambda /(T\times T))\ge 1$, as claimed.

Next we show that for generic $g\in G$, $(H\times K)_g$ contains a nontrivial torus. Suppose not. Then for generic $g\in G$, $(H\times K)_g^0$ is a unipotent subgroup of H of dimension at least 2, so $(H\times K)_g^0$ is a maximal unipotent subgroup of H and has dimension 2. But then the orbit $(H\times K)\cdot g$ is dense in G, so $G/(H\times K)$ is a single point, which is a contradiction.

It follows from the proof of the positive-dimensional case of Theorem 9.1 that $(H\times n_iKn_1^{-1})\cdot L_\lambda$ is dense in G for some nontrivial $\lambda \in Y(T)$ and some i. But $N_G(T)$ normalizes K, so $(H\times K)\cdot L_\lambda$ is dense in G (and hence $L_\lambda$ is a short-root Levi subgroup of G). We deduce from the discussion above that generic $(H\times K)$-orbits in G are closed. Moreover, we see that the map $L_\lambda /(T\times T)\rightarrow G/(H\times K)$ is finite and dominant, hence surjective. A simple calculation shows that generic stabilizers of $T\times T$ on $L_\lambda$ have dimension 1, so $\mathrm{dim}(L_\lambda /(T\times T))= 1$, which implies that $\mathrm{dim}(G/(H\times K))= 1$. Hence generic stabilizers of $H\times K$ on G are reductive groups of dimension 3. It follows that for generic $g\in G$, $(H\times K)_g^0$ is of type $A_1$.

(b). Let $p= 2$ and let $H= K= B$. Then generic orbits of $H\times K$ on G are closed and for generic $g\in G$, $(H\times M)_g^0$ is of type $A_1$. The proof is similar to case (a).

(c). Let $p= 2$, let $H= A$ and let $K= B$. Consider the stabilizer $(H\times K)_1$, which is isomorphic via the map $\phi _1$ to $H\cap K$. It is easily seen that $H\cap K= T$, so $(H\times K)_1= \{(t,t)\mid t\in T\}$. It follows that $\mathrm{dim}((H\times K)\cdot 1)= \mathrm{dim}(H\times K)- \mathrm{dim}(T)= 12- 2= 10= \mathrm{dim}(G)$, so $(H\times K)\cdot 1$ is dense in G and generic stabilizers have dimension 2 and are reductive. It follows from Theorem 9.1 that generic orbits are closed. Hence $(H\times K)\cdot 1$ is closed and $H\times K$ acts transitively on G. (This conclusion also follows from [14, Theorem  A], since A and B are maximal connected subgroups of G.)

We finish the section with a further example.

Example 9.12

Suppose $p\ne 2$ and let G be simple and of rank r. Let $\tau \in \mathrm{Aut}(G$) be an involution that inverts a maximal torus of G—such a $\tau$ always exists, by [17, Lemma 3.6]—and let $H= K= C_G(\tau )$. Then $(H\times K)_g= H\cap gKg^{-1}$ is a finite group of order $2^r$ for generic $g\in G$ [17, Theorem 9], so G has good dimension for the $H\times K$-action, by Theorem 9.1. (Note that in [17, Sect.  3.2] one considers the action of H by left multiplication on G / K rather than the double coset action of $H\times K$ on G, but the arguments carry over easily to our setting. See also [37, Example 8.4].)

We now consider a striking feature of this example: namely that, although generic stabilizers for the double coset action are nontrivial, there is a unique $(H\times K)$-orbit O consisting of elements with trivial stabilizer [17, Theorem  9] (compare Example 3.2). If $p= 0$ then O cannot be closed: for otherwise there would exist an étale slice through any element of O, so every element in some open neighbourhood of O would have trivial stabilizer by the argument of Example 3.2, a contradiction. More generally, for arbitrary $p\ne 2$ the argument of [37, Ex. 8.4] implies that G has a principal stabilizer A which is a finite group of order $2^r$, and it follows from Theorem 1.1 that if $g'\in G$ and $(H\times K)\cdot g'$ is closed then $(H\times K)_{g'}$ contains a conjugate of A. Hence we see again that O cannot be closed.

We give a direct proof of this. The orbit O is of the form $(H\times K)\cdot g$, where $g\in G$ has the property that $u:= \tau \tau ^g$ is a regular unipotent element of G and u is inverted by $\tau$ (see [17, Proposition 3.1]). In fact, we can choose g to be a regular unipotent element of G such that $g^2= u$ and $\tau$ inverts g (take g to be $u^s$ if $p> 0$, where $2s\equiv 1\ \mathrm{mod}\ |u|$). Set $U= \overline{\langle g\rangle }$; then $\tau$ normalizes U, as $\tau$ inverts g. There exists $\lambda \in Y(G)$ such that $\lim _{a\rightarrow 0} \lambda (a)g'\lambda (a)^{-1}= 1$ for all $g'\in U$. We can choose $\lambda$ to be optimal in the sense of [9, Defn. 4.4 and Theorem  4.5] (cf. Sect. 2.5). Then $\tau$ normalises $P_\lambda$. Now $N_{\mathrm{Aut}(G)}(P_\lambda )$ is an R-parabolic subgroup of the reductive group $\mathrm{Aut}(G)$ [35, Proposition 5.4(a)], so $N_G(P_\lambda )= P_\mu$ for some $\mu \in Y(G)$. As $\tau \in P_\mu$ and $\langle \tau \rangle$ is linearly reductive, we can choose $\mu$ to centralize $\tau$: that is, we can choose $\mu$ to belong to Y(H).

Let $\sigma = (\mu , \mu )\in Y(H\times K)$. Then $\lim _{a\rightarrow 0} \sigma (a)\cdot g= \lim _{a\rightarrow 0} \mu (a)g\mu (a)^{-1}= 1$ since $U\le R_u(P_\lambda )= R_u(P_\mu )$. But clearly $1\not \in O$, so O is not closed, as claimed.

10 Applications to G-complete reducibility

We finish with some applications of ideas from Sects. 3 and 9 to G-complete reducibility. Our next lemma gives a basic structural result about G and its subgroups which can quickly be proven using the framework we have now set up; the setting is as in Sect. 9 but more general, since we allow one of the subgroups to be non-reductive (cf. [15]). The argument used is taken from the proof of [29, Kap. III.2.5, Satz 2]; note that although the reference [29] works with groups and varieties defined over the complex numbers, many of the arguments are completely general. For convenience, we reproduce the details here.

Lemma 10.1

Suppose K is a subgroup of G and let H be a reductive subgroup of G that contains a maximal torus of K. Then HK is a closed subset of G.

Proof

First suppose that K is unipotent. The quotient $X=G/H$ is affine and H is the stabilizer in G of the point $x = \pi _{G,H}(1) \in X$. Since K is unipotent, and all orbits for unipotent groups on affine varieties are closed [10, Proposition 4.10], $K\cdot x$ is closed, so KH (and hence HK) is closed in G by Lemma 3.5.

Now, in the general case, let T be a maximal torus of K contained in H and let B be a Borel subgroup of K containing T with unipotent radical U. Then $UH = BH$ is closed in G by the first paragraph, and the following argument from [29, Kap. III.2.5, Satz 2] gives us what we want. We have a sequence of morphisms

$$\begin{aligned} K\times G \overset{\phi }{\longrightarrow } K\times G \xrightarrow {\rho =\pi _{K,B} \times \mathrm{id}} K/B \times G \overset{\mathrm{pr}_2}{\longrightarrow } G \end{aligned}$$

where $\phi (g',g):=(g',g'g)$ for $g' \in K$, $g \in G$, $\pi _{K,B}$ is the quotient morphism $K \rightarrow K/B$ and $\mathrm{pr}_2$ is the projection of $K/B \times G$ onto the second factor. Let $Y = K \times BH$. Since BH is closed in G, Y is closed in $K \times G$. Since $\phi$ is an isomorphism of varieties, $\phi (Y)$ is closed in $K\times G$ and therefore $\rho (\phi (Y))$ is closed in $K/B \times G$. Finally, since K / B is complete, $\mathrm{pr}_2(\rho (\phi (Y)))$ is closed in G. But it is easy to see that $\mathrm{pr}_2(\rho (\phi (Y)))=KH$, so we are done.

$\square$

The lemma above allows a quick proof of the following result.

Proposition 10.2

Suppose G is reductive, X is an affine G-variety and $x \in X$. If H is a reductive subgroup of G containing a maximal torus of $G_x$, then $H\cdot x$ is closed in $G\cdot x$. In particular, if $G\cdot x$ is closed in X then $H\cdot x$ is closed in X.

Proof

By Lemma 10.1, under the given hypotheses, $HG_x$ is closed in G. Hence, by Lemma 3.5, $H\cdot x$ is closed in $G\cdot x$. $\square$

Remark 10.3

The argument of Lemma 10.1 is used in [29, Kap. III, 2.5, Folgerung 3] to show that if X is affine and $G_x$ contains a maximal torus of G, then $G\cdot x$ is closed, a result which has obvious similarities to Proposition 10.2.

Corollary 10.4

Suppose H and K are reductive subgroups of the reductive group G. If $H\cap K$ contains a maximal torus of H or K, then HK is closed in G and $H\cap K$ is a reductive group.

Proof

Without loss, suppose $H\cap K$ contains a maximal torus of K. The first conclusion is a special case of Lemma 10.1. For the second, apply Proposition 10.2 to the action of G and H on the quotient G / K. Since G / K is affine and $H\cdot \pi _{G,K}(1)$ is closed in G / K, this orbit is also affine and hence the stabilizer $H_{\pi _{G,K}(1)}= H\cap K$ is reductive by Lemma 3.3(ii). $\square$

A theme running through [5] and subsequent papers on complete reducibility by the same authors is the following general question: if A and H are subgroups of G with $A\subseteq H$ and H reductive, what conditions ensure that if A is G-cr then A is H-cr, and vice versa? Because of the link between complete reducibility and closed orbits in $G^n$ explained in Sect. 2.4 above, this is readily seen to be a special case of the general questions considered in this paper. Since this was one of the original motivations for the work presented here, we briefly record some of the translations of our main results into the language of complete reducibility and give a couple of other consequences in this setting.

First note that Proposition 10.2 specializes to [5, Proposition 3.19] in the setting of complete reducibility: that is, with notation as just set up, if H also contains a maximal torus of $C_G(A)$ and A is G-cr, then A is H-cr. More generally, we have:

Proposition 10.5

Suppose H is a reductive subgroup of G, and let A be a subgroup of H.

• (i) If A is H-cr, then $HC_G(A)$ is closed in G.

• (ii) If A is G-cr, then A is H-cr if and only if $HC_G(A)$ is closed in G.

Proof

(i). Let ${\mathbf {a}} \in H^n$ be a generic tuple for A. Suppose A is H-cr; then $H\cdot {\mathbf {a}}$ is closed in $H^n$. Since $H^n$ is closed in $G^n$, $H\cdot {\mathbf {a}}$ is closed in $G\cdot {\mathbf {a}}$. Therefore, by Lemma 3.5(i), $HG_{\mathbf {a}} = HC_G(A)$ is closed in G.

(ii). Using a generic tuple for A again, this becomes a direct application of Lemma 3.5(ii). $\square$

The notions of reductive pairs from [46, Sect. 3] and separability from [5, Def. 3.27] have proved useful in the study of complete reducibility: see [3, 5, 7, Sect. 3.5] for example. Recall that the definitions of reductive pair and separable subgroup were given in Sect. 7. We have the following result:

Proposition 10.6

Suppose (G, H) is a reductive pair. Let A be a separable subgroup of G contained in H. Then $HC_G(A)$ is closed in G.

Proof

Let ${\mathbf {a}} \in H^n$ be a generic tuple for A. Then $C_G(A) = G_{\mathbf {a}}$, and since A is separable in G, the orbit $G\cdot {\mathbf {a}}$ is separable. Now Richardson’s “tangent space argument” [46, Sect. 3] (generalized to n-tuples in [53]) shows that $G\cdot {\mathbf {a}} \cap H^n$ decomposes into finitely many H-orbits, each of which is closed in $G\cdot {\mathbf {a}} \cap H^n$. Since one of these orbits is $H\cdot {\mathbf {a}}$, we can conclude that $H\cdot {\mathbf {a}}$ is closed in $G\cdot {\mathbf {a}} \cap H^n$, and hence in $G\cdot {\mathbf {a}}$. Therefore, $HG_{\mathbf {a}} = HC_G(A)$ is closed in G by Lemma 3.5(i). $\square$

Remark 10.7

Note that every pair (G, H) of reductive groups with $H \le G$ is a reductive pair in characteristic 0 and the separability hypothesis is also automatic. In characteristic $p>0$, every subgroup of G is separable as long as p is “very good” for G; see [7, Theorem  1.2].

As a final remark, we note that there are Lie algebra analogues of Propositions 10.5 and 10.6, where we replace the subgroup A with a Lie subalgebra of ${{\,\mathrm{Lie}\,}}(H)$. For details of how to make such translations, see [9, Sect. 5], for example.