I'm physics student learning mathematical physics, now especially Bessel functions.
I learned that the Bessel functions and that family are derived from the ODE $$ y''+y'/x+(1-\nu^2/x^2)y=0, $$ and $J_\nu(x)$, $Y_\nu(x)$, $H_\nu^{(1)}(x)$, $H_2^{(2)}(x)$, $I_\nu(x)$, $K_\nu(x)$ have similar properties to $\cos(x)$, $\sin(x)$, $e^{ix}$, $e^{-ix}$, $\cosh(x)$, $\sinh(x)$. (I'm using the notation in the Arfken's Mathematical methods for physicists.)
So I naturally began to wonder if we can also generalize the ODE for trigonometric function $y''+y=0$, to depend on certain index like $\nu$, so we can generate special functions which depend on the index, and which becomes trigonometric function for certain index. But I'm not sure where should I put the index $\nu$ to make family of functions. If I set, like, $y''+\nu^2y=0$, then solution is just $\sin(\nu x)$, which is basically same as $\sin(x)$. Wait... but maybe we can do this way, $y''+(1-\nu^2/x^2)y=0$?
Is there a extension of trigonometric functions in this(or similar) way? Are they just not practical functions? Or is there a reason that trigonometric functions are special so they should be unique?
There isn't really a single unique possible extension, because there's a myriad ways in which you might deform the trigonometric functions, and it comes down to the human subjective judgement on whether the deformation is useful enough to be classified as a special function or not.
However, there is a pretty clear answer in the Jacobi elliptic functions $\operatorname{sn}(x,k)$, $\operatorname{cn}(x,k)$, and related objects; these are explained in depth in Wikipedia and the Digital Library of Mathematical Functions. Since you explicitly mentioned the second-order ODE, I'll quote one relevant differential equation $$ \frac{{\mathrm{d}}^{2}}{{\mathrm{d}z}^{2}}\operatorname{sn}\left(z,k\right) = -(1+k^{2})\operatorname{sn}\left(z,k\right) +2k^{2}{\operatorname{sn}^{3}}\left(z,k\right), $$ (with more found here), but that isn't really the cleanest way to obtain that family in a natural way; instead, the first place they tend to pop up is as inverses of the elliptic integrals.