Question:
Locate all relative maxima, relative minima and saddle points (if any) for the function
$$f(x,y)=y\sqrt{x} - y^2-2x+7y$$
My attempt:
$$f_x(x,y)=\frac{y}{2\sqrt{x}}-2$$
$$f_y(x,y)=\sqrt{x}-2y+7$$
For the critical point $(x_0, y_0)$, $f_x(x_0,y_0)=0$ and $f_y(x_0,y_0)=0$. Following this line of reasoning we can find one critical point $(x_0,y_0)=(1,4)$. I will not expand upon that.
My interest is in $f_x(x,y)$ being undefined. Have a look at this definition from Anton:
13.8.5 Definition A point $(x_0, y_0)$ in the domain of a function $f(x, y)$ is called a critical point of the function if $f_x(x_0, y_0) = 0$ and $f_y(x_0, y_0) = 0$ or if one or both partial derivatives do not exist at $(x_0, y_0)$.
$f_x(x,y)$ does not exist for $x=0$. So, let our second critical point be $(0, k)$. Now, how do I find the value of $k$? It seems to me that $(0, k)$ represents an infinite number of points. So, does the above function have an infinite number of critical points?
It seems like you've already answered your own question. As you point out yourself, any point $(0,y)$ with $y\neq0$ (to avoid $0/0$) lies in the domain of $f$ and leads to nonexistence of the partial derivative with respect to the first argument. Hence, such a point satisfies the definition of a critical point you stated. (Whether all those points are relevant for your problem is a different story.)