problem: Find the smallest set $A$ s.t. $f:\mathbb{R}^3\setminus A\to \mathbb{R}^2$ defined by $f(x,y,z)=(x^2y+z, 2xy-z^2)$ is a submersion.
my attempt: since $df|_p(v)=(2xydx+x^2dy+dz, 2ydx+2xdy-2zdz)|_p(v)=(2p_1p_2dx|_p(v)+p_1^2dy|_p(v)+dz|_P(v), 2p_2dx|_p(v)+2p_1dy|_p(v)-2p_3dz|_p(v))$,
where $p=(p_1,p_2,p_3)\in \mathbb{R}^3$ and $v\in T_p\mathbb{R}^3(\cong\mathbb{R}^3)$. (so $dx|_p(v), dy|_p(v),dz|_p(v)$ are real numbers)
we need to find $v$ s.t. $df|_p(v)=(a,b)\in\mathbb{R}^2$ for all $(a,b)\in\mathbb{R}^2$ since $df|_p:T_p\mathbb{R}^3\cong \mathbb{R}^3\to T_{f(p)}\mathbb{R}^2\cong \mathbb{R}^2$, and for $f$ to be submersion, $df$ should be surjective.
now let $v=\alpha \frac{\partial}{\partial z}|_p+\beta\frac{\partial}{\partial y}|_p+\gamma \frac{\partial}{\partial z}|_p\in T_p\mathbb{R}^3$. since $dx_i|_p(\frac{\partial}{\partial x_j}|_p)=\delta_{ij}$ where $x_1=x, x_2=y, x_3=z$, we have
$df|_p(v)=(2p_1p_2\alpha+p_1^2\beta+\gamma, 2p_2\alpha+2p_1\beta-2p_3\gamma)=(a,b)$.
so from above equation, we need to find a smallest set of $p$ s.t. $\alpha, \beta, \gamma$ don't exist.
using a matrix notation, it becomes
$\begin{bmatrix} 2p_1p_2 & p_1^2 & 1\\ 2p_2 & 2p_1 & -2p_3\end{bmatrix} \begin{bmatrix}\alpha\\\beta\\\gamma\end{bmatrix}= \begin{bmatrix}a\\b\end{bmatrix}$ $\dots$ (1)
let $F=\begin{bmatrix} 2p_1p_2 & p_1^2 & 1\\ 2p_2 & 2p_1 & -2p_3\end{bmatrix}$ (and I noticed that $F=J|_p(f)$, the Jacobian matrix evaluated at $p$, so maybe there was some shortcut to begin with this omitting all the computations above... I mean, is $df|_p=J|_p(f)$ always true?)
(*) if $F$ has a left inverse, then $\alpha, \beta, \gamma$ exist, $F$ becomes surjective, $f$ becomes a submersion.
so I guess we have to go through a tedious matrix computation inevitably:
find $E$ s.t. $EF=\begin{bmatrix}e_1 & e_2\\ e_3 & e_4 \\ e_5 & e_6\end{bmatrix}\begin{bmatrix} 2p_1p_2 & p_1^2 & 1\\ 2p_2 & 2p_1 & -2p_3\end{bmatrix}=I_3=\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1\end{bmatrix}$
we have 6 unknowns and 9 equations. so this is solvable in terms of $e_i$'s and doing so we can get the smallest condition for $p$.
but according to proposition 4(1)(i) of this article, such $E$ doesn't exist; here my confusion starts.
and assuming $df|_p=J|_p(f)(=F)$ is true, for $df|_p$ to be surjective, $F$ should have right inverse since surjection iff having right inverse. according to my argument, I was trying to find a left inverse $E$ of $F$, not right inverse(*). but I don't know still why I was wrong; multiply a left inverse(it doesn't exist though) both sides of (1) then we get coefficients of $v$, so $F$ satisfies the condition of surjection. I guess I need some linear algebra.
and I actually tried to solve $EF=I_3$ and got a contradiction; such $E$ didn't exist.
then, now, we need to find a right inverse $G$ s.t. $FG=I_2$, but $G$ contains 6 unknowns and we have only 4 equations in this case. I tried to find some conditions for $p$ by solving these equations but failed.
so, finally, my questions are these:
1) is $df|_p=J|_p(f)$ always true?
2) why is (*) wrong?
3) then how to solve this problem?
$f$ is a submersion if $\Bbb d f$ has maximum rank (i.e. $2$) at every point. In order to compute the rank, it is useful to put $\Bbb d f$ in matrix form as
$$\Bbb d f = \begin{pmatrix} 2xy & x^2 & 1 \\ 2y & 2x & -2z \end{pmatrix} .$$
There are three minors of order $2$, and we want that at least one of them be $\ne 0$. Putting it differently, let us find the points at which all the minors are $0$, and then look at the complementary.
We get the system (the $2$ on the second row does not change things, so we drop it)
$$\left\{ \begin{eqnarray} x^2 y = 0 \\ x^2z + x = 0 \\ xyz + y = 0 \end{eqnarray} \right.$$
the analysis of which is not difficult.
If $x = 0$ in the 1st equation, then the first two equations are satisfied, and the 3rd implies $y=0$, therefore the solutions are $(0,0,z)$.
If $y = 0$ in the 1st equation, then the 1st and 3rd equation are satisfied and there are again two cases given by the 2nd equation:
2.1 $x=0$, so again we get the solution $(0,0,z)$;
2.2 $x \ne 0$, so $xz = -1$, so the solution is $(x, 0, - \frac 1 x)$.
We get that $f$ is not a submersion at the points of $A = \{(0, 0, z) \mid z \in \Bbb R\} \cup \{(x, 0, \frac 1 x) \mid x \ne 0\}$. Taking the complementary of this, you get the subset on which $f$ is a submersion.
What does it mean to find the "smallest" $A$? Well, if to the above $A$ you add any other subset, $f$ will still be a submersion on the complementary of this newly obtained larger set. The procedure above gave us directly the smallest $A$.