Usually, whenever the Laplace equation is used as a model for a physical phenomenon (say in a 3D Cartesian domain), we are quick to assume that the solution takes the form $u(x,y,z) = X(x)Y(y)Z(z)$. This is what is meant by the solution being separable. I don't remember ever encountering a problem where a non-separable solution is considered.
Are there nontrivial solutions to the Laplace equation that are not separable?
Is there a proof that non-trivial, non-separable solutions exist? A reference would be nice!
Have analytical non-separable solutions ever been found in a physics problem? Examples would be nice!
What branches of mathematics concern questions of what kinds of solutions exist to a differential equation? Or more practically (for me), what courses at a university would cover this kind of question?
It is certainly not true that every solution to the Laplace equation is separable; the sum of two different separable solutions generally won't be, for example. But the Laplace equation is linear, so we don't need to find every solution, we only need to find enough solutions that the solutions we care about can be written as (possibly infinite) linear combinations. When we make this ansatz what we're hoping is that the physically relevant solutions are (possibly infinite) linear combinations of separable solutions.
As a concrete example, let's consider the Laplace equation $\Delta f = 0$ for a function $f(x, y)$ of two variables, with no boundary conditions (ignoring how physically realistic or unrealistic this might be). Solutions to this equation are known as harmonic functions, and they can be identified with the real parts of holomorphic functions on the complex plane $\mathbb{C}$. For example, the function $z^2$ is holomorphic; writing $z = x + iy$ we have $(x + iy)^2 = (x^2 - y^2) + 2i xy$ so its real part is
$$f(x, y) = x^2 - y^2.$$
You can easily verify that this is a harmonic function but it is not separable; it cannot be written as $g(x) h(y)$ for any functions $g, h$ (this is a nice exercise, to figure out how to prove this).
Depending on what exactly you're interested in, the relevant branches of mathematics might be partial differential equations or functional analysis.