Introduction

It is a great privilege to contribute to Henry Stapp’s 90th birthday Festschrift. My interactions with Henry go back to the 1970s when I was starting out on my exploration of quantum mechanics. I joined David Bohm at Birkbeck College, London, in 1961, beginning a collaboration that lasted for thirty-five years. In those early years, we did not discuss his famous 1952 papers (Bohm 1952) on ‘hidden variables’ which eventually morphed into ‘Bohmian mechanics’, the latter was an approach that we had both rejected from its inception, instead outlining an alternative which we summarised in The Undivided Universe (Bohm and Hiley 1993). We were exploring a more radical idea that Bohm called ‘structure process’ (Bohm 1965). The aim was to provide a fundamentally new approach to quantum phenomena which we hoped would ultimately lead to a theory of quantum gravity.

We were joined by Roger Penrose who at the time was in the Birkbeck mathematics department, developing a theory of spin networks and twistor theory (Penrose 1967, 1971). Naturally, our discussions centred around the question of how to unite QM and GR. In this environment, my introduction to quantum theory was unorthodox to say the least!

The approaches of both Bohm and Penrose abandoned the idea of assuming an a priori given space-time in which fields and particles evolve in time. Instead, we were exploring possible deeper structures from which space-time itself would emerge as some form of coarse-grained approximation. Penrose showed how the classical rotation group emerged from a quantum spin structure when large quantum numbers were involved. It is this structure that now forms the basis of quantum loop gravity, a subject that has recently developed rapidly into the type of theory that we had in mind way back in the 1960s but had failed to make any headway at that stage.

But I get ahead of myself. In the 1960s, we first discussed a possibility of describing ‘structure process’ in terms of an underlying simplicial structure using a discrete de Rham cohomology to provide a link to quantum numbers (Bohm et al. 1970). However, the model was too static emphasising more the ‘structure’ at the expense of the ‘process’. I then discovered the work of Benn and Tucker (1987), who showed how differential forms used in the de Rham approach could be generalised to link with the abstract generators of Clifford algebras. The vital piece of background to this algebra was the work of Clifford himself. He had started from the idea of process, which depended crucially on the order of action. Remarkably, Clifford (1882) was working, in those pre-quantum days, entirely within classical physics, yet discovered an algebra that now plays a key role in quantum mechanics when spin and relativity are introduced. In light of this, we introduced the notion of the ‘algebra of process’. This provided a key link with Penrose’s twistors which were, of course, the semi-spinors of the conformal Clifford algebra (see Bohm and Hiley 1984).

The Emergence of the Classical World

Gentle Photons

It was at this time that we became aware of the ideas that Henry and his colleague, Geoff Chew, were exploring at Berkeley. There was a common theme. We all agreed that quantum theory in the hands of Bohr offered a set of rules for calculating the statistical results under well-defined experimental conditions. However, it did not provide an ontology that would unambiguously remove the observer from playing an essential role in the process. Henry proposed that it should be the electromagnetic field that provided the link between classical and quantum properties, a link in which the twistor was to play a role. My preference is for gravity to play such a role since it permeates everything but that is for the future.

Following the ideas discussed in an early paper of Bloch and Norsdsieck (1937), Henry suggested that the ‘infrared catastrophe’, rather than being a problem, should be used in a positive way to provide a method to separate out the classical aspect of the total process. In this way, he (Stapp 1983) completely solved the technical problem and showed that it was the coherent state of the electromagnetic field that replaced the ‘observer’ in Bohr’s approach. Recall that coherent states have classical-like solutions in phase space. In the case of the em field, the expectation value in the coherent state |A(t)〉 of the quantum operator \(\hat A(x)\) corresponding to the vector potential can be written as

$$ \begin{array}{@{}rcl@{}} A(x)=\langle A(x^{\prime})|\hat A(x)|A(x^{\prime})\rangle. \end{array} $$

Further work shows that the S-matrix can be expressed in terms of A(x) establishing the presence of the classical electromagnetic field in the theory with its position in space-time, rather than configuration space. Furthermore, the incorporation of light into the S-matrix automatically brings into this description an exact classical level that is coordinated to the ordinary four-dimensional space-time continuum of special relativity.

By this means, the structure process can provide a well-ordered sequence of actual events in space-time so that it is meaningful to regard each quantum process as a sequence of actual events in space-time. Thus, our ontology contains no explicit dependence on human observers. In this sense, the ideas discussed in this paper are different from the position Henry now favours. For example, the Feynman path can be considered as an actual sequence of events in space-time, totally independent of human intervention. This enables us to show exactly how the Bohm approach that I had worked on in the seventies fits into the standard approach to quantum mechanics (Flack and Hiley 2018).

The Bohm Approach

In the 1970s, I was encouraged by two of our research students, Chris Philippidis and Chris Dewdney, to examine in more detail Bohm’s 1952 papers (Bohm 1952). The titles of these papers contained the phrase ‘hidden variables’, an approach that many thought had failed in its aims, including myself and I had never taken it seriously. In spite of this, we decided to use the rapidly developing new computer technology to calculate ‘trajectories’, examine their form and look at the detailed structure of the quantum potential [see Eq. 2 below] for various characteristic quantum phenomena such as two-slit interference, barrier penetration, scattering by square wells etc.

The approach was simple, take the classical canonical relations p = ∇S and E = −∂tS, replace the classical action by the phase of the wave function and calculate ‘trajectories’. Amazingly, we found that the results gave a powerful intuitive picture of what could be going on provided our assumptions were correct. The key question was why was this approach apparently working so well?

It was much later when I noticed that Dirac (1947) had introduced an algebraic approach that turns out to be the forerunner of the Bohm approach (Bohm 1952), producing exactly the same equations that Bohm used. However, Dirac argued that proceeding in this way the existence of a local momentum would violate the uncertainty principle. Bohm, in contrast, showed that this was not true. Furthermore, he realised that one could still retain classical ideas by using the first-order WKB approximation, but keeping all the terms of the expansion demanded a radical change of outlook. It was those terms that summed into the quantum potential, a notion that Heisenberg regarded as ad hoc (Heisenberg 1958). However, it was the forerunner of deformation quantum mechanics (Bayen et al. 1978).

Dirac went on to suggest that the Lagrangian played a key role in his approach and so I took the Lagrangian that Heisenberg had used to ‘derive’ the Schrödinger equation. Rewriting the Lagrangian using the polar decomposition of the wave function \(\psi =Re^{iS/\hbar }\) one finds that the Euler-Lagrange equations give the two equations that Bohm had used, namely,

$$ \begin{array}{@{}rcl@{}} \frac{\partial P}{\partial t}+\nabla.\left( P\frac{\nabla S}{m}\right)= 0 \end{array} $$
(1)

and

$$ \begin{array}{@{}rcl@{}} \frac{\partial S}{\partial t} +\frac{(\nabla S)^{2}}{2m} +Q +V = 0 \end{array} $$
(2)

where \(Q=-\frac {\hbar ^{2}}{2mR}(\nabla ^{2} R)\) is the expression for the quantum potential. But this still does not explain why the relation p = ∇S works.

Once we have the Lagrangian, we can use the general expression for the energy-momentum tensor to find

$$ \begin{array}{@{}rcl@{}} T^{0\mu} = \frac{i}{2}[\psi^{*} {\overset{\leftrightarrow}{\partial}}^{\mu} \psi]=-\rho\partial^{\mu} S. \end{array} $$

Explicitly, p = ∇S = T0j/ρ and E = −∂tS = T00/ρ. So the Bohm momentum actually emerges from the energy-momentum tensor derived from the Schrödinger Lagrangian. But there is more. The trace of the energy-momentum is

$$ \begin{array}{@{}rcl@{}} (T^{kk}-\mathcal{L}\delta^{kk})/\rho=\frac{(\partial^{k}S)^{2}}{2m}+\frac{(\partial^{k}R)^{2}}{2mR^{2}} +V. \end{array} $$

Thus, not only does the kinetic energy, KEB = p2/2m = (∇S)2/2m, emerge but there also appears a new form of kinetic energy, namely, KEO = (∇R)2/2mR2. This latter is clearly connected with the appearance of the quantum potential. It turns out that I had rediscovered some early work of Takabayasi (1954).

I found all this very reminiscent of early work of Feynman (1948) and Schwinger (1951, 1953a, b) when they laid the foundations of quantum field theory. Schwinger (1953b) argued that the fundamental quantum dynamical laws would find their proper expression in terms of transition amplitudes [TAs], not in terms of the Schrödinger wave functions. The key ingredient in his work was the energy-momentum tensor which he used to define momentum TA. In the non-relativistic theory, the momentum TA is simply

$$ \begin{array}{@{}rcl@{}} \langle \hat P^{\mu}\rangle=\frac{\langle \phi(x,t)|\hat P^{\mu}| \psi(x_{0},t_{0})\rangle}{ \langle \phi(x,t)|\psi(x_{0},t_{0})\rangle} \end{array} $$
(3)

an expression that is exactly the same as the weak value of the momentum, about which we will have more to say later.

It was Dirac (1945) who suggested that, in the non-relativistic case, we should divide the connection between two states ψ(x0,t0) and ϕ(x,t) into a series of infinitesimal time steps 𝜖 = tj+ 1 − tj, enabling us to construct a ‘path’ out of a series of TAs so that

$$ \begin{array}{@{}rcl@{}} \langle x_{t}|x_{t_{0}}\rangle=\int \langle x_{t}|x_{j}\rangle dx_{j}\langle x_{j}|x_{j-1}\rangle\dots\langle x_{2}|x_{1}\rangle dx_{1}\langle x_{1}|x_{t_{0}}\rangle \end{array} $$
(4)

where the 〈xj+ 1|xj〉 are a set of infinitesimal TAs. In this way, as Dirac argues (Dirac 1945), we can discuss trajectories for the motion of a quantum particle, which makes quantum mechanics more closely resemble classical mechanics. Indeed, the method enables one to bring out the close analogy between classical and quantum contact transformations, an analogy that Bohm highlights in his book, Quantum Theory (Bohm 1951).

It is interesting to note that in sections 31 and 32 of his book, Dirac (1947) derived Eqs. 1 and 2 from an algebraic point of view but did not pursue the approach because he thought the uncertainty principle would be violated. Bohm showed that this conclusion was not correct by developing his causal approach.Footnote 1 All this prompts the question ‘Is there a relation between the Feynman paths and the Bohm trajectories?’

Feynman Paths and Bohm Trajectories

In Eq. 4, one writes 〈x|x′〉 = exp[iS(x,x′)], where Feynman assumes \(S(x,x^{\prime })=\delta \int \mathcal {L}(x, x^{\prime })dt\), the classical action and therefore

$$ \begin{array}{@{}rcl@{}} S(x,x^{\prime})=\frac{m(x-x^{\prime})^{2}}{2\epsilon} \end{array} $$

𝜖 being a small time interval. The momentum TA at a point X between x′ and x is

$$ \begin{array}{@{}rcl@{}} p_{X}(x,x^{\prime})&=&\frac{\partial S(X, x^{\prime})}{\partial X}+\frac{\partial S(x, X)}{\partial X}\\&=&\left[\frac{(X-x^{\prime})}{\epsilon}-\frac{(x-X)}{\epsilon}\right]. \end{array} $$

Notice that the derivative is not continuous at X. Instead, we have a ‘backward derivative’ (X − x′)/𝜖 and a ‘forward derivative’ (x − X)/𝜖 at X. Thus, the Feynman path is continuous but nowhere differentiable.

Over time an ensemble of individual particles pass through X, so that there is a distribution of momenta arriving at X and a distribution of momenta leaving the point. Thus at each point X, we have an average value of the momentum and that average value must be determined by the wave function. The average momentum at a point turns out to be the Bohm momentum p = ∇S, S being the phase of the wave function.

To see how this result follows, consider the process shown in Fig. 1. This gives an image of an ensemble of actual individual quantum events occurring at X together with the incoming and outgoing sprays of momenta. Thus, we have two contributions to consider, one spray coming from the point x′ and the other spray leaving for the point x. We must determine the distribution of momenta in each spray to produce a result that is consistent with the wave function ψ(X) at X. Therefore, we can write

$$ \begin{array}{@{}rcl@{}} \lim\limits_{x^{\prime}\rightarrow X}\psi(x^{\prime})&=&\int\phi(p^{\prime})e^{ip^{\prime}X} dp^{\prime}\quad\text{and} \\\lim\limits_{X\rightarrow x}\psi^{*}(x)&=&\int \phi^{*}(p)e^{-ipX}dp. \end{array} $$
Fig. 1
figure 1

Enfolding and unfolding at a point

Full size image

The ϕ(p′) contains information regarding the probability distribution of the incoming momentum spray, while ϕ∗(p) contains information about the probability distribution in the outgoing momentum spray. These wave functions must be such that in the limit 𝜖 → 0 they are consistent with the wave function ψ(X).

Thus, we can define the mean momentum, \(\overline {\overline {P}}(X)\) as

$$ \begin{array}{@{}rcl@{}} \rho(X)\overline{\overline {P}}(X)&=&\iint P\phi^{*}(p)e^{-ipX}\phi(p^{\prime})e^{ip^{\prime}X}\\&&\times \delta(P-(p^{\prime}+p)/2)dPdpdp^{\prime} \end{array} $$
(5)

where ρ(X) is the probability density at X. We have added the restriction δ(P − (p′ + p)/2) since momentum is conserved at X. We can rewrite Eq. 5 and form

$$ \begin{array}{@{}rcl@{}} \rho(X)\overline{\overline {P}}(X) = \frac{1}{2\pi}\iint P\phi^{*}(p + \theta/2)e^{-iX\theta}\phi(p - \theta/2)d\theta dP \end{array} $$

or equivalently taking Fourier transforms

$$ \begin{array}{@{}rcl@{}} \rho(X)\overline{\overline {P}}(X)&=&\frac{1}{2\pi}\iint P\psi^{*}(X-\sigma/2)e^{-iP\sigma}\\&&\times \psi(X+\sigma/2)d\sigma dP \end{array} $$

which means that \(\overline {\overline {P}}(X)\) is the conditional expectation value of the momentum weighted by the Wigner function. Equation 5 can be put in the form

$$ \begin{array}{@{}rcl@{}} \rho(X)\overline{\overline {P}}(X)=\left( \frac{1}{2i}\right)[(\partial_{x_{1}}-\partial_{x_{2}})\psi(x_{1})\psi(x_{2})]_{x_{1}=x_{2}=X} \end{array} $$
(6)

an equation that appears in the Moyal approach (Moyal 1949), which is based on a different but isomorphic non-commutative algebra. If we evaluate this expression for the wave function written in polar form ψ(x) = R(x)exp[iS(x)], we find \(\overline {\overline P}(X)=\nabla S(X)\) which is just the Bohm momentum.

Since the Bohm momentum is an average value, the trajectories calculated from them must be averages, so that each Bohm ‘trajectory’ is an average of an ensemble of individual Feynman paths. It is not the momentum of a single ‘particle’ passing the point X, as assumed in Bohmian mechanics, but the mean momentum flow at the point in question.

Weak Values

The previous section was motivated by the appearance in 2011 of an experiment reporting the construction of ‘photon trajectories’ using a two-slit optical setup that measured the weak values of the transverse momentum (Kocsis et al. 2011). This was all made possible by utilising a general idea introduced by Aharonov et al. (1988) who defined the weak value of any operator \(\hat A\) by

$$ \begin{array}{@{}rcl@{}} \langle \hat A\rangle=\frac{\langle \phi(t)|\hat A |\psi(t_{0})\rangle}{ \langle \phi|(t)|\psi(t_{0})\rangle}. \end{array} $$

Clearly, Eq. 3 is a special case of a weak value. Indeed, Leavens (2005) and Wisemen (2007) argued that the weak value of the momentum provided, in principle, a way of experimentally constructing Bohm trajectories. It was this connection that Kocsis et al. (2011) exploited in their two-slit optical experiment. Their results were remarkably similar to the Bohm trajectories (Philippidis et al. 1979). (For a detailed comparison, see Coffey and Wyatt 2011.) These experimental results show clearly the statistical nature of the results used to construct the flow lines, thus confirming the statistical nature of the Bohm trajectories.

There is a problem for the claimed interpretation of flow lines in that photons, per se, do not have trajectories. Nevertheless, Flack and Hiley (2016) showed that what Kocsis et al. (2011) had constructed were mean momentum flow lines using the real part of the weak Poynting vector. If one requires a more appropriate comparison to the Bohm trajectories then one should experiment using atoms. In fact, our group at UCL are at present measuring weak values of momentum using argon and helium (Morley et al. 2016, 2017) and the experiments are very close to completion.

The Algebra of Process

The Technical Details

It should by now be apparent that the Bohm approach has its origins in the non-commutative algebra pioneered by Born et al. (1926) and which I have called ‘the algebra of process’ (Hiley 1980). The key link appears in Dirac’s The Principles of Quantum Mechanics (Dirac 1947) . There a symbol, 〉, the ‘standard ket’ is introduced into the algebra. This enables the wave function ψ to be written as a wave operator, \(\psi (\hat Q,\hat P) \rangle \); technically, the new object turns the wave function into an element of a left ideal. To complete the algebra, a dual symbol, the standard bra is introduced. Thus, Dirac has replaced the usual Hilbert space by a non-commutative algebra formed by the symbols \([\hat Q,\hat P, \epsilon ]\) where the 𝜖 =〉〈, an idempotent.Footnote 2 In this way, all the essential details of the quantum process are contained in the algebra.

In effect, what Dirac has done is to introduce a new representation, which I call the Dirac-Bohm picture (Hiley and Dennis 2018). This representation is unitarily equivalent to the Schrödinger picture and supplements the Heisenberg, Interaction and Fock pictures. It is different in that the unitary transformation is based on the action rather than the energy. Indeed, 𝜖 plays an analogous role to the vacuum projector in the Fock representation.

Using this approach, I was able to propose two time evolution equations within the algebra (Hiley 2015), equations implicitly contained in Dirac’s work (Darrigol 1992). A similar pair of equations can also be constructed in the Moyal algebra. I have used these equations to extend the Dirac-Bohm picture to Clifford algebras and shown how the Pauli and Dirac equations fit into the structure (Hiley and Callaghan 2012). Contrary to the usual assumption, there is no problem with extending this picture to include relativity. In passing, I should also like to mention that this approach was inspired by Penrose’s development of twistors. It is this algebra that provides a possible approach to quantum gravity.

The Overarching Philosophy

The theme of this paper was motivated by the paper that Henry kindly submitted to Bohm’s 70th Festschrift (Hiley and Peat 1987). There, Henry talks about the ‘Bohm-Heisenberg idea of events’. I agree on the notion of quantum events that actualise, but I wanted to clarify how Bohm’s 1952 paper (Bohm 1952) related to the Heisenberg non-commutative algebra. The Bohm approach emerges as a coarse-grained average, giving the appearance of a deterministic approach, but being, in fact, very different from classical determinism.

Our approach restores the position-momentum symmetry and so two views emerge, a phase space constructed from (x,p = ∇xS(x)) or from (x = −∇pS(p),p). These are the shadow phase spaces. In the Dirac picture, they correspond to choosing the idempotent defined by \(\hat P\epsilon _{x} = 0\) or \(\hat X\epsilon _{p} = 0\).Footnote 3

Bohr proposed that we understand this dual view through the principle of complementarity, a philosophical principle that does not sit comfortably with physicists in general. Bohm proposed a new notion of the implicate-explicate order. The need for non-commutativity suggests that we can no longer provide one unique, God’s-eye, view of natural phenomena. Because we are inside, as it were, we can only project out partial views determined by the experimental conditions which enable us to construct particular shadow manifolds, or explicate orders. The underlying reality is implicate. Bohm investigated the consequences of this implicate view of reality, not only in physics but also in other areas of intellectual discourse. The one that would most interest here is the application of these ideas to mind, but I do not have the space to discuss this here. Have a happy 90th Henry!