Let $Y$ be a square and nonsingular matrix (invertible).
Given $X \prec 0$, if we multiply $Y^*$ on the left and $Y$ on the right to the $X$ (assuming the dimension matches),
I have a claim that
$$ X \prec 0 \iff Y^*XY \prec 0 $$
Question: I can get an intuitive understanding with a scalar analogy but I am not sure how I can prove this.
Attempt:
I first tried transforming this into the quadratic form (making it scalar).
For instance, multiply them by the vectors $z^*$ and $z$, left and right, respectively:
$$ X \prec 0 \implies \forall z \neq 0, \quad z^*Xz < 0. $$ By letting $z = Yw$ for non zero $w$ ($z^{*} = w^{*}Y^{*}$), $$ \implies \forall w \neq 0, \quad w^*Y^*XYw < 0 \implies Y^*XY \prec 0.$$
For the opposite direction, we have:
$$ \forall w \neq 0, \quad w^*Y^*XYw < 0, $$
Let $z = Yw$ and since $Y^{-1}$ exsits by the assumption, we have $w = Y^{-1}z$.
By substituting $w$ into the above inequality:
$$ \forall z \neq 0, \quad z^*Xz < 0 \implies X \prec 0.$$
Is my proof correct?
Some side question I have:
Is the assumption of $Y$ being square and nonsingular essential? I obviously used it in my attempt above. If I don't have it, is it possible still to get the same result?
Your proof is right. For the side question, you need $Y$ to be square and invertible to show that the relation is iff.
Example for $\implies$: let $X=-1$ and $Y=[1 \ 1]$, then you have $X<0$, but $$Y^\top X Y=\begin{bmatrix} -1 & -1 \\ -1 & -1\end{bmatrix}\leq 0$$
Example for $\Longleftarrow$: let $X=\begin{bmatrix} -1 & -1 \\ -1 & -1\end{bmatrix}$ and $Y=\begin{bmatrix} 1\\ 1 \end{bmatrix}$, then $Y^\top X Y=-4<0$, but $X\leq 0$.