Suppose I have a function $f(x)=||Ax-b||_2^2$. So, its hessian is as follows $\nabla^2(x)=2A^TA$ which is positive definite if $A\ge0$ which means $f(x)$ is convex. But how to determine its convexity if there is additional restriction like $f(x) = ||Ax-b||_2^2/(1-x^Tx)$ where $||x||_2\le1$. My understanding is as $x^Tx$ is a scalar we can take $(1-x^Tx)=c$, where $c$ is also scalar. So, $f(x)=c||Ax-b||_2^2$ is convex given $c$ is defined. But the retriction on $x$ makes me confused. As $||x||_2$ can be equal to 1, so $c\to\infty$ in which case $f(x)$ is no longer convex?
Update: As pointed out the comment, in the 1d case, we have $f(x)=(ax-b)^2/(1-x^2)$, taking $a=1$ and $b=0$, and plotting for the value for the valid range of $|x|\le1$, it looks like a convex function to me!
Getting pretty confused. I am also pretty new to this topic and so I can make some silly mistakes. But I'll be really grateful if someone can clarify. Thanks.