$$\frac{\sin \lambda \alpha } {\sin \alpha} - \frac{\cos \lambda \alpha}{\cos \alpha} = \lambda - 1$$ where $\alpha$ cannot be integral multiple of $\frac{\pi}{2}$.
Playing with the LHS I ended up with following equation $$\frac{2\sin \{\alpha(\lambda - 1 )\}}{\sin 2\alpha } = \lambda -1$$ after that i don't how to solve this trig equation.
On
By inspection, $\lambda = 1$ is a solution. When $\lambda \ne 1,$ we have $2\sin(\lambda-1)x = (\lambda-1)\sin(2x)$. Taking the Taylor series of both sides, $$ 2(\lambda-1)x - 2\frac{(\lambda-1)^3x^3}{3!} + O(n^5)= (\lambda-1)(2x)- (\lambda-1)\frac{2^3x^3}{3!}+O(n^5)$$ These must be equal term-by-term. The first terms are identical. Comparing the second terms, we see that $(\lambda-1)^2 = 4 \implies \lambda-1 = \pm 2\implies \lambda \in \{-1,3\}.$ We easily check that these give solutions, so the only solutions are $\boxed{ \lambda \in \{-1,1,3\}}$
On
For something somewhat more empirical, consider $\alpha = \frac{\pi}{4}$. For this value, your derived equation reduces to
$$ 2\sin \frac{(\lambda-1)\pi}{4} = \lambda-1 $$
The below graph of the LHS and RHS as a function of $\lambda$ shows that there are possible solutions at $-1$, $1$, and $3$, and nowhere else.
For $\lambda = -1$, we have
$$ \frac{\sin -\alpha}{\sin \alpha} - \frac{\cos -\alpha}{\cos \alpha} = -2 $$ $$ -1 - 1 = -2 $$
which works. For $\lambda = 1$, we have
$$ \frac{\sin \alpha}{\sin \alpha} - \frac{\cos \alpha}{\cos \alpha} = 0 $$ $$ 1-1 = 0 $$
which again works. And finally, for $\lambda = 3$, we have
$$ \frac{\sin 3\alpha}{\sin \alpha} - \frac{\cos 3\alpha}{\cos \alpha} = 2 $$ $$ 2 \cos (2x) + 1 - [2 \cos (2x) - 1] = 2 $$
which works once more.
On
It helps to write the equation in this form.
$$ \frac{\sin[\alpha(\lambda-1)]}{\alpha(\lambda - 1)} = \frac{\sin(2\alpha)}{2\alpha} $$
Simply put, we're looking for $f(\alpha(\lambda-1))=f(2\alpha)$ where $f(x) = \dfrac{\sin x}{x}$. Everything from here on will involve the sinc function.
Here's a plot
If $\alpha \ne 0$, there are three solutions:, $\lambda = 1$ and $\lambda-1 = \pm 2$ or $\lambda = -1, 3$.
A special case occurs when $\sin (2\alpha) = 0$, or $\alpha = \dfrac{n\pi}{2}$, $n \ne 0$. Then you get rational solutions $\lambda = 1 + \dfrac{2m}{p}$, where $m$ is an integer and $p$ is any factor of $n$. This solution set is countably infinite.
EDIT: You don't want $\alpha = \dfrac{n\pi}{2}$ since this makes the original equation undefined, but everything below this still holds.
For any other case, you'll always get a finite number of solutions, since the sinc function decays in amplitude. Obtaining a definite count is possible, but rather complicated, since it requires knowing the extrema of the sinc function. You'll have to use numerical methods from here.
It's easy to count the solution set if $f'(2\alpha) = 0$. Then there are no solution $|\lambda - 1| > 2$.
and so on. There are a total of $2n+1$ solutions (including $\lambda = 1$) for $|2\alpha| \in \big(n\pi, (n+1)\pi\big)$
If $f'(2\alpha) \ne 0$, then there may exist some $\beta$ such that $f'(\beta) = f'(2\alpha)$ and $f(\beta) = f(2\alpha)$. If this happens, there are once again $2n+1$ solutions for $|\beta| \in \big(n\pi, (n+1)\pi\big)$
Finally, if $f(2\alpha)$ does not coincide with any extrema, then it has to occur between two maxima or minima. We can then find a $c$ such that $f'(c) = 0$ and $|f(2\alpha)-f(c)|$ is minimal. Then there are $2n-1$ solutions for $|c| \in \big(n\pi, (n+1)\pi\big)$
Tldr; the problem boils down to counting the solutions of $\dfrac{\sin x}{x} = a$
By inspection $\lambda=3 $ for all values of $\alpha$. $ \lambda = 1,-1 $ are also solutions by inspection.