$f(x,y,z)=\frac{(y+2z)^2}{(x-3y)}$,$\quad$ on $\{(x,y,z)\in R^3\mid x-3y>0\}$.
I tried to figure out the Hessian matrix of $f(x,y,z)$ to see whether it is positive definite , but it's very complex. Is there any easy way or any trick to do with this problem?
Any help would be appreciate.
First, prove the convexity of this simpler function: $$g:\mathbb{R}^2\rightarrow\mathbb{R}, \quad g(x,y) \triangleq x^2/y, \quad \mathop{\textrm{dom}} g = \{(x,y)\,|\,y>0\}$$ Now define the affine function $$h:\mathbb{R}^3\rightarrow\mathbb{R}^2, \quad h(x,y,z) = (y+2z,x-3y)$$ Then $f$ is just the composition of $g$ and $h$; that is, $f=g\circ h$. The composition of a convex outer function and an affine inner function is always convex.
Frankly, it is rare to prove convexity from first principles (that is, the derivative test or the secant test) unless that function is very simple. You should be using rules such as those outlined in Chapter 3 of Boyd & Vandenberghe to save you some trouble.