Waves are unlike particles in three important ways. Firstly, a wave spreads out while a particle is confined to a small area. Secondly, a classical particle follows a single trajectory, whereas a wave can propagate simultaneously along many paths. Thirdly, two waves can pass through each other, while two particles coming together suffer a collision.
In 1923 American physicist Arthur Holly Compton discovered another experimental occurrence of Einstein's “particles of light”. Compton shone X-rays onto a target containing loosely bound electrons (famously graphite), and was able to detect both the ejected electron and the particle of light that recoiled in the manner of the cue ball in snooker (the “recoil photon”).
Then in 1924 French aristocrat Louis de Broglie submitted a PhD thesis to the Sorbonne that proposed a theory of electron waves and predicted a wave-particle dual nature of matter. His thesis professor was not convinced, but sent a copy to Einstein, who backed the idea, and de Broglie passed his PhD. De Broglie was pointing towards a fundamentally new theory of physics. Classical physics had reduced the world to matter and fields, or particles and waves. Planck, Einstein and Compton had all provided reasons why waves must sometimes be thought of as particles, and now de Broglie was saying particles can sometimes be thought of as waves. At this point nobody had filled in all of the mathematical details, and nobody had a clue how reality could be made of stuff that was simultaneously a wave and a particle. Then in 1925 not one but three quantum theories arrived.
Werner Heisenberg represented a quantum system with a set of matrices, so his theory is called matrix mechanics. In matrix mechanics, physical quantities such as position or momentum are represented by matrices. The diagonal elements correspond to possible measurement outcomes, while the off-diagonal elements encode transitions or couplings between states. The matrices have diagonal entries that represent the probability that the system has that specific value, and off-diagonal entries that represent the strength of the quantum connections between possible values for that attribute. So in this system the momentum, position and other attributes of a quantum entity such as an electron is represented by one of these matrices, rather than by a single number.
Erwin Schrödinger represented a quantum system as a wave form and wrote quantum laws of motion, of the sort that waves must obey, which is known as the Schrödinger equation. His theory is called wave mechanics.
Paul Dirac's theory is harder to explain. Dirac showed that quantum states could be represented as vectors in an abstract space, and that different mathematical formulations (matrix mechanics, wave mechanics) were simply different coordinate representations of the same underlying structure.
Heisenberg: reality is a discrete set of "jumps.
Schrödinger: reality is a smooth, continuous field (until measurement).
Dirac: reality is an abstract geometric state.
The mathematics is clear, but the meaning is a mess. Two things are important to note from all this. The first is that quantum theory is purely mathematical, and the mathematics can be represented in several different ways. The second is that probability is inherent in this mathematics.
With the mathematics completed, and three versions available, each of which can be used in different situations, it was now possible for other scientists and engineers to start using quantum theory for all sorts of scientific and technological work, which would eventually lead to the development of nuclear weapons and power. But there was still a huge unanswered question about the implications of the theory for our understanding of the nature of reality. How can something be a wave and a particle at the same time? What are we to make of the probabilistic nature of the mathematics?
It is often claimed that unless you have a deep understanding of the mathematics, it is impossible to understand the philosophical relevance of quantum theory. I do not have this level of understanding. However, if the claim was actually true then we should expect our greatest mathematicians to arrive at converging conclusions that would eliminate the mystery, which is very obviously not happening. It follows that the differences in interpretation must be the result of the understanding – or rather the misunderstanding – of something other than the mathematics. The truth is that quantum theory does not, on its own, supply any conclusive answers to the intriguing philosophical questions it poses.
Between 1925 and 1935 Bohr and Einstein debated the meaning of quantum theory, and a patchy, incomplete consensus emerged for an interpretation favoured by Bohr, Heisenberg and Max Born. This became known as the Copenhagen Interpretation (CI) (Bohr's institute was in Copenhagen) and it remains the most popular interpretation today, regardless of its many faults. The CI is non-realist or instrumentalist: it denies that quantum theory describes an underlying reality. You can be forgiven for wondering what the hell that is supposed to mean, and many other physicists objected, Einstein among them. Surely, they said, this goes too far: it is premature to conclude that no future technology could reveal a deeper truth, and all we should say is that, for now, we'll be cautious and sceptical. But Bohr was having none of it. “There is no quantum world,” he said, “there is only an abstract quantum description.” Heisenberg took a similar view: “The hope that new experiments will lead us back to objective events in time and space is about as well founded as the hope of discovering the end of the world in the unexplored regions of the Arctic.” In other words, the CI is a forthright attempt to shut down questions about the ultimate nature of reality. “Quantum theory is unbelievably weird, but we've nailed the maths and you're never going to get any deeper answers so you might as well stop asking all those awkward questions. Move along please; there is nothing to see here.”
The originator of the quantum himself harboured deep metaphysical convictions that ran counter to the emerging instrumentalist consensus. Max Planck, the reluctant revolutionary, came to believe that the quantum revolution pointed not to a lack of reality, but to a different kind of foundation. In his later years, he argued unequivocally for the primacy of mind: “I regard consciousness as fundamental. I regard matter as derivative from consciousness... Everything that we talk about, everything that we regard as existing, postulates consciousness.” For Planck, the quantum was not a reason to stop asking about reality, but a signpost pointing toward a mental, or ideal, ground of being – a path the Copenhagen school would deliberately choose not to follow.
The questions did not go away. For all the Copenhagenists' vehement denials that quantum theory can tell us anything about the nature of reality, their own interpretation raises very specific questions. There are two parts to the CI. The first is that there is no reality in the absence of observations, and the second is that observation somehow creates reality (in practice, CI treats measurement as the point at which quantum possibilities become definite, though Bohr himself avoided saying that consciousness literally creates reality). The problem is that if that's your theory then it is rather important to explain exactly what “observation” means, and the CI doesn't even try. In the words of physicist Murray Gell-Mann in 1976: “Niels Bohr brainwashed a whole generation of physicists into thinking the job was done fifty years ago.”
Next section: The Uncertainty Principle | Two-Phase Cosmology