I have a bunch of points $[x_n, y_n, z_n]$ and I want to find the angles $\vartheta_s$, $\vartheta_t$ and $\vartheta_y$ that minimize
\begin{equation} \begin{split} &\sum_{n=0}^{N} (\, (\cos{\vartheta_y}\left[ x_n\cos{\vartheta_s}-y_n\sin{\vartheta_s} \right]\\ &-\sin{\vartheta_y}\left[ \cos{\vartheta_t} \left(x_n\sin{\vartheta_s}+y_n\cos{\vartheta_s}\right) -z_n\sin{\vartheta_t} \right])\cdot k_1\\ & + (\,\sin{\vartheta_y}\left[ x_n\cos{\vartheta_s}-y_n\sin{\vartheta_s} \right]\\ &+\cos{\vartheta_y}\left[ \cos{\vartheta_t} \left(x_n\sin{\vartheta_s}+y_n\cos{\vartheta_s}\right) -z_n\sin{\vartheta_t} \right]\,)\cdot k_2\, )^2 \end{split} \end{equation}
where $k_1$ and $k_2$ are known scalars.
Now, my first idea was to differentiate with respect to the three angles, which would give me three equations that I can set equal to zero. If I then group the terms to separate my three angles, I can get a matrix notation and invert it and solve for the three angles. However, after differentiating, I am not able to split my equations into three terms $\vartheta_t*...$, $\vartheta_y*...$, and $\vartheta_s*...$.
Are there any tricks to deal with trigonometric problems like this?
There are infinitely many solutions. You can minimize the sum by the following steps:
Calculate $\kappa$ from $k_1, k_2$ such that $$\cos{\kappa} = \frac{k_1}{\sqrt{k_1^2+k_2^2}}$$ $$\sin{\kappa} = \frac{k_2}{\sqrt{k_1^2+k_2^2}}$$
Calculate the matrix $\mathbf{D}$ from the data $x_n, y_n, z_n$: $$ \mathbf{D} = \left[ \begin{matrix} \sum{x_n x_n} & \sum{x_n y_n} & \sum{x_n z_n} \\ \sum{x_n y_n} & \sum{y_n y_n} & \sum{y_n z_n} \\ \sum{x_n z_n} & \sum{y_n z_n} & \sum{z_n z_n} \end{matrix} \right] $$
Calculate (analytically (see here e.g.) or numerically) $\lambda_3$, the smallest eigenvalue of $\mathbf{D}$, and the corresponding unit eigenvector $\mathbf{u_3}$ with components $$ \mathbf{u_3}= \left[ \begin{matrix} u_x \\ u_y \\ u_z \end{matrix} \right] $$
Choose any value for $\vartheta_s$.
Calculate $\vartheta_y-\kappa$ and $\vartheta_t$ from: $$ \cos{(\vartheta_y-\kappa)} = u_x \cos{\vartheta_s} - u_y \sin{\vartheta_s} \\ u_z = \sin{(\vartheta_y-\kappa)} \sin{\vartheta_t} $$
A more detailed explanation follows.
Write the sum of the squares $Z$ as
$$Z=\sum_{n=0}^{N} {t_n^2}$$
where
$$\begin{equation}\begin{aligned} t_n &= k_1 \cdot (\cos{\vartheta_y}\left[ x_n\cos{\vartheta_s}-y_n\sin{\vartheta_s} \right] -\sin{\vartheta_y}\left[ \cos{\vartheta_t} \left(x_n\sin{\vartheta_s}+y_n\cos{\vartheta_s}\right) -z_n\sin{\vartheta_t} \right])\\ &+ k_2 \cdot (\sin{\vartheta_y}\left[ x_n\cos{\vartheta_s}-y_n\sin{\vartheta_s} \right] +\cos{\vartheta_y}\left[ \cos{\vartheta_t} \left(x_n\sin{\vartheta_s}+y_n\cos{\vartheta_s}\right) -z_n\sin{\vartheta_t} \right]) \end{aligned}\end{equation}$$
Rearrange to get $$\begin{equation}\begin{aligned} t_n &= (k_1 \cdot \cos{\vartheta_y} + k_2 \cdot \sin{\vartheta_y}) \left[ x_n\cos{\vartheta_s}-y_n\sin{\vartheta_s} \right] \\ &-(k_1 \cdot \sin{\vartheta_y} - k_2 \cdot \cos{\vartheta_y})\left[ \cos{\vartheta_t} \left(x_n\sin{\vartheta_s}+y_n\cos{\vartheta_s}\right) -z_n\sin{\vartheta_t} \right] \end{aligned}\end{equation}$$
Now define $\kappa$ such that $$\cos{\kappa} = \frac{k_1}{\sqrt{k_1^2+k_2^2}}$$ $$\sin{\kappa} = \frac{k_2}{\sqrt{k_1^2+k_2^2}}$$
Then $$\begin{equation}\begin{aligned} \frac{t_n}{\sqrt{k_1^2+k_2^2}} &= (\cos{\kappa} \cdot \cos{\vartheta_y} + \sin{\kappa} \cdot \sin{\vartheta_y}) \left[ x_n\cos{\vartheta_s}-y_n\sin{\vartheta_s} \right] \\ &-(\cos{\kappa} \cdot \sin{\vartheta_y} - \sin{\kappa} \cdot \cos{\vartheta_y})\left[ \cos{\vartheta_t} \left(x_n\sin{\vartheta_s}+y_n\cos{\vartheta_s}\right) -z_n\sin{\vartheta_t} \right] \end{aligned}\end{equation}$$
Or $$\begin{equation}\begin{aligned} \frac{t_n}{\sqrt{k_1^2+k_2^2}} &= \cos{(\vartheta_y-\kappa)} \left[ x_n\cos{\vartheta_s}-y_n\sin{\vartheta_s} \right] \\ &-\sin{(\vartheta_y-\kappa)}\left[ \cos{\vartheta_t} \left(x_n\sin{\vartheta_s}+y_n\cos{\vartheta_s}\right) -z_n\sin{\vartheta_t} \right] \end{aligned}\end{equation}$$
So the impact of the scalars $k_1$ and $k_2$ can be confined to an additive offset of $\vartheta_y$.
Rearrange again to this form $$\begin{equation}\begin{aligned} \frac{t_n}{\sqrt{k_1^2+k_2^2}} &= x_n \left[\cos{(\vartheta_y-\kappa)} \cos{\vartheta_s}-\sin{(\vartheta_y-\kappa)} \sin{\vartheta_s} \cos{\vartheta_t} \right]\\ &+ y_n \left[-\cos{(\vartheta_y-\kappa)} \sin{\vartheta_s} - \sin{(\vartheta_y-\kappa)} \cos{\vartheta_s} \cos{\vartheta_t} \right]\\ &+ z_n \left[\sin{(\vartheta_y-\kappa)} \sin{\vartheta_t} \right] \end{aligned}\end{equation}$$
Then write a column column vector of the $t_n$'s
$$\begin{equation}\begin{aligned} \frac{1}{\sqrt{k_1^2+k_2^2}} \mathbf{t} &=\left[ \begin{matrix} \mathbf{x}&\mathbf{y}&\mathbf{z} \end{matrix} \right] \mathbf{\Phi} \\ &=\mathbf{X} \mathbf{\Phi} \end{aligned}\end{equation}$$
where $\mathbf{x}$, $\mathbf{y}$ and $\mathbf{z}$ are column vectors of $x_n$, $y_n$ and $z_n$, respectively, and $$ \mathbf{\Phi}= \left[ \begin{matrix} \cos{(\vartheta_y-\kappa)} \cos{\vartheta_s}-\sin{(\vartheta_y-\kappa)} \sin{\vartheta_s} \cos{\vartheta_t} \\ -\cos{(\vartheta_y-\kappa)} \sin{\vartheta_s} - \sin{(\vartheta_y-\kappa)} \cos{\vartheta_s} \cos{\vartheta_t} \\ \sin{(\vartheta_y-\kappa)} \sin{\vartheta_t} \end{matrix} \right] $$
Then $$\begin{equation}\begin{aligned} Z &= \mathbf{t^T} \mathbf{t} \\ &= (k_1^2 + k_2^2) \mathbf{\Phi^T} \mathbf{X^T} \mathbf{X} \mathbf{\Phi} \\ &= (k_1^2 + k_2^2) \mathbf{\Phi^T} \mathbf{D} \mathbf{\Phi} \\ \end{aligned}\end{equation}$$
where $$ \mathbf{D} = \mathbf{X^T} \mathbf{X} = \left[ \begin{matrix} \sum{x_n x_n} & \sum{x_n y_n} & \sum{x_n z_n} \\ \sum{x_n y_n} & \sum{y_n y_n} & \sum{y_n z_n} \\ \sum{x_n z_n} & \sum{y_n z_n} & \sum{z_n z_n} \end{matrix} \right] $$
It can be shown that $\mathbf{\Phi^T \Phi} \equiv 1$ so $\mathbf{\Phi}$ is a unit vector. Further, $\mathbf{D}$ is a symmetric, positive definite matrix with positive eigenvalues $\lambda_1 \ge \lambda_2 \ge \lambda_3$ and corresponding unit eigenvectors $\mathbf{u_1}$, $\mathbf{u_2}$ and $\mathbf{u_3}$ that are mutually orthogonal. From this we observe that $Z$ is minimized by setting $\mathbf{\Phi} = \mathbf{u_3}$, the eigenvector corresponding to the smallest eigenvalue:
$$ \mathbf{\Phi}= \left[ \begin{matrix} \cos{(\vartheta_y-\kappa)} \cos{\vartheta_s}-\sin{(\vartheta_y-\kappa)} \sin{\vartheta_s} \cos{\vartheta_t} \\ -\cos{(\vartheta_y-\kappa)} \sin{\vartheta_s} - \sin{(\vartheta_y-\kappa)} \cos{\vartheta_s} \cos{\vartheta_t} \\ \sin{(\vartheta_y-\kappa)} \sin{\vartheta_t} \end{matrix} \right]= \left[ \begin{matrix} u_x \\ u_y \\ u_z \end{matrix} \right]=\mathbf{u_3} $$
This system of three simultaneous equations can be solved by multiplying the first by $\cos{\vartheta_s}$ and the second by $\sin{\vartheta_s}$ and subtracting to give
$$ \cos{(\vartheta_y-\kappa)} = u_x \cos{\vartheta_s} - u_y \sin{\vartheta_s} $$
There are infinitely many solutions. Choose any value for $\vartheta_s$. Then calculate $\cos{(\vartheta_y-\kappa)}$ by the previous equation. Finally calculate $\vartheta_t$ from $u_z = \sin{(\vartheta_y-\kappa)} \sin{\vartheta_t}$.
The minimal value for $Z$ is $Z_{min}=\lambda_3 (z_1^2 + z_2^2)$.