I am trying to find the critical points of the equation:
$$f(x,y) = x\ln y + x^2 -4x -5y +8 .$$
Found the partial derivative in terms of $x$ and got $\ln y +2x -4$ and for $y$ I got $\frac{x}{y} -5$.
I equated the two to $0$:
$$\ln y +2x -4 = 0 , $$
$$\frac{x}{y} -5 = 0 .$$
How do I find $x$ and $y$?
From the second equation, $y=\frac x5$. Plug in the first to obtain $$2x+\log(x)=4+\log(5)$$ that is to say $$2x+\log(2x)=4+\log(10)\implies x=\frac{1}{2} W\left(10 e^4\right)$$ where $W(.)$ is Lambert function. Numerically, $$x \sim 2.37270 \implies y\sim 0.47454$$
If you do not want to (or cannot) use Lambert function, just use graphing or inpection; this shows the the solution is $\in (2,3)$. Make one iteration of Newton method : $$x_0=2\implies x_1=2+\frac{2}{5} \log \left(\frac{5}{2}\right) \sim 2.36652$$ $$x_0=3\implies x_1=\frac{3}{7} \left(5+\log \left(\frac{5}{3}\right)\right) \sim 2.36178$$