Suppose that I want to solve the Helmholtz equation, $$ \bigg[\nabla^2 +k^2\bigg]\psi(\mathbf r)=0 $$ in the three-dimensional conical domain $z>\sqrt{x^2+y^2}$, with Dirichlet conditions $\psi(x,y,\sqrt{x^2+y^2})=0$ on the boundary, and that I want to do this via separation of variables in an orthogonal coordinate system that respects the geometry of my domain. I suspect that this is doable via some form of hyperbolic coordinates where one family of coordinate surfaces are hyperboloids that limit to the cone, and another coordinate is the azimuthal angle about the apex, but I'm having some trouble identifying exactly which coordinate system this is.
So, I want to ask:
- what orthogonal-coordinates system is best suited to this geometry?
- what do the forwards and backwards coordinate transformations look like?
- is the Helmholtz equation separable on this system? if so, what does it look like for each coordinate?
- are the corresponding eigenfunctions known in terms of existing special functions? if so, which ones?
- finally, and if possible, are any vector Helmholtz eigenfunctions known in this geometry?
In spherical coordinates the Laplacian separates as $\Delta=\partial_r^2+\frac{1}{r}\partial_r+\frac{1}{r^2}\Delta_{S^2}$, thus you can compute its eigenfunctions in terms of spherical harmonics (i.e. the eigenfunctions of $\Delta_{S^2}).$ To satisfy Dirichlet conditions on the boundary of the cone $C$ you are looking for those spherical harmonics which are zero on $C\cap S^2$.
For the vector valued case one can derive a similar expression for the Laplacian (using $\Delta_{S^2}$), but assuming that you have some background on Riemannian geometry, let me show you how your question translates into a well studied setting: Consider $\Omega = C\cap S^2$ as Riemannian manifold with metric $\bar g$ inherited from the sphere, then $\Omega \times (0,\infty)$ with the metric $dr^2+r^2\bar g$ is isometric to $C$. (So far this is just a fancier way to talk about spherical coordinates.) In this paper (§3) the author gives formulas for the eigenfunctions of the Hodge-Laplacian of spaces with this kind of metric. The Hodge-Laplacian on $1-$forms is dual to the vector valued Laplacian.