Given that $\Psi : R^3 \to R^3$ is defined by $\Psi (r,\alpha, x_3)=(r \cos \alpha, r \sin \alpha ,x_3),$ I am trying to find a maximal open set $V$ in $R^3$ such that $\Psi : V \to \Psi(V)$ is a $C^{\infty}$ diffeomorphism.
My attempt finding a maximal open set $V$:
Because
$$ \det D\Psi (r,\alpha , x_3)=0 \ \text{if}\ r=0,$$ there is no diffeomorphism between sets that contain a point of the form $(0,0,x_3).$ So, the maximal set equals either
$$ V = \{ (r,\alpha,x_3)\ | \ 0 < r < \infty, 0 < \alpha < 2 \pi , -\infty < x_3 < \infty \} $$ or $$V = \{ (r,\alpha,x_3)\ | \ 0 < r < \infty, 0 \leq \alpha < 2 \pi , -\infty < x_3 < \infty \}. $$
I would greatly appreciate any feedback.
I believe you have done ok.
We need to compute the Jacobian. So $$J=\begin{pmatrix} \cos\alpha & \sin\alpha & 0\\ -r\sin\alpha &r\cos\alpha&0\\ 0&0&1\end{pmatrix}$$.
The determinant is $\det J=r\not =0$ when $r\not =0$.
Now as to the open set I would choose your first set, because it appears to be open... the second set does not...