Assume (homogeneous) system of linear equations in $X$ of the form $Q\operatorname{vec}(X)=0$, where $Q$ is in general a tall non-square matrix. I want to find a non-trivial matrix $X$ (i.e. $X \neq 0$). A possible solution is given by constraining matrix $X$ to have unitary norm, i.e., $\|X\|_F^2=1$. A possible solution is given by a right eigenvector of the matrix $Q$ associated to a zero singular value.
But how can I enforce other kind of constraints, for instance that $\operatorname{det}(X) \neq0$? Do I have any closed form solution like the SVD above?
It is impossible to fulfill the requirements in general. We will see this in the following example.
Let $Q$ be the matrix below: $$ Q=\begin{bmatrix}1& 0&0&0\\ 0&1&0&0\\ 0&0&0&1\\ 1&1&0&1\\ 1&1&0&1 \end{bmatrix}. $$ The three rows are important information. The rest of the rows are to make the matrix tall. Let $\textrm{vec} X = \begin{bmatrix} x_1 \\ x_2 \\ x_3\\ x_4 \end{bmatrix}$. Then $Q \textrm{vec}X=0$ gives $x_1=x_2=x_4=0$ and $x_3$ is free.
We have $$ X=\begin{bmatrix} 0 & x_3 \\ 0 & 0 \end{bmatrix} $$ will always have $\det X=0$.