I have a system in which an unknown $x \in \mathbb{R}^4$ is measured throught a set of variables $y \in \mathbb{R}^6$. Specifically, $y=Ax$ where each entry of $A$ is an extremely long algebraic expression of trigonometric functions of two unknown variables $\alpha$ and $\beta$, $0\leq \alpha,\beta \leq \pi/4$. I confirmed (numerically) that the singular values of A are non zero for all $\alpha$ and $\beta$. This assures that I can compute $x$ from $y$ once I know $\alpha$ and $\beta$. In my case, however, $\alpha$ and $\beta$ are unknown.
How can I check if given a certain $y$ the possible $\alpha,\beta$ is unique without sampling the whole 6 dimensional space of $y$?
Define $u^T := \begin{bmatrix} \alpha & \beta & x_1 & x_2 & x_3 & x_4 \end{bmatrix}$. Then $$y = f(u) := A(\alpha,\beta)x$$ Now you can use nonlinear least squares or other iterative methods to solve for $u$. For example, start with an estimation $u_0$ and use it to solve the linear equation (possibly by using pseudo-inverse) $$ y = \frac{\partial f(u)}{\partial u}|_{u=u_0} u $$ and use this new $u$ in the equation again until solutions converge.
Note that uniqueness or existence of the solution is not guaranteed. Because $y$ may not be in the image of $A$ for any $\alpha, \beta$ depending on the formula of the entries of $A$. The described method may find a solution in the vicinity of $u_0$, i.e. a local minimum.
Edit: There is not enough information to determine if a solution exists or if it is unique. For a counter example let's start with a simplified problem like $$\begin{bmatrix}y_1 \\ y_2 \end{bmatrix} = \begin{bmatrix}a_{11}(\alpha) \\ a_{21}(\alpha) \end{bmatrix} x_1$$ For a unique solution to exist for a given $y$, when you draw a line in the direction of $y$, it must cut exactly one point of the parametric plot of $a_1(\alpha)$ (that is the first - and only in this case - column of $A$) geometrically.
So, for example $$\begin{bmatrix}-1 \\ 1 \end{bmatrix} = \begin{bmatrix}\sin^2(\alpha) \\ \cos^2(\alpha) \end{bmatrix} x_1$$ does not have any solution and if it describes a (sufficiently complex) Lissajous curve you don't have a unique solution.