Here's what I tried:
- Found the partial derivatives: $fx = 3x^2y - 2xy^4$ and $fy = x^3 - 4x^2y^3$
- Plugging in $(5, -4)$, I ended up with $(-6700, -6275)$ for $fx$ and $fy$, then took the square root of those numbers squared to get $sqrt(84265625)$, which probably means I'm doing something wrong.
I'm new to this, so I'm trying to follow an example which took the partial derivatives, found a vector by plugging in the point to those partial derivatives, found the "length" of the vectors, and divided the vectors by the length. Then it multiplied it by the vectors. This probably doesn't make much sense, but I was trying to follow the example here: http://www.math.wisc.edu/~benguria/qs4.pdf
The change in $f$ at $(5,-4)$ in the direction of the unit vector $\hat u$ is $\nabla f|_{(5,-4)}\cdot\hat u=|\nabla f|_{(5,-4)}|\hat{u}|$cos$\theta=|\nabla f|_{(5,-4)}$cos$\theta$, where $\theta$ is the angle between $\nabla f|_{(5,-4)}$ and $\hat u$. This is maximum at $\theta=0$ and minimum at $\theta =\pi$. In other words, maximum in the direction of $\nabla f|_{(5,-4)}$ and minimum in the direction opposite of $\nabla f|_{(5,-4)}$
So what's $\nabla f|_{(5,-4)}$? It's the vector $(\frac{\partial f}{\partial x}(5,-4),\frac{\partial f}{\partial y}(5,-4))=(3x^2y-2xy^4,x^3-4x^2y^3)|_{(5,-4)}=(-2860,6525)$. Note this is different from what you got, so one of us did the algebra wrong. We normalize $\nabla f|_{(5,-4)}$ by dividing by the norm $|\nabla f|_{(5,-4)}|=\sqrt{f_x^2(5,-4)+f_y^2(5,-4)}=5\sqrt{2030209}.$ Finally, the direction of steepest descent is opposite of $|\nabla f|_{(5,-4)}|$, so we multiply this by $-1$ to obtain $\hat{u}:=-\frac{1}{5\sqrt{2030209}}(-2860, 6525)$ as the direction of steepest descent.