I found $F_x=3x^2+10yx$ and $F_y=5x^2+3y^2$, then $D_uf=\frac{9}{5}x^2+6yx+4x^2$
$F_{x2}= \frac{58}{5}x + 6y, F_{y2}=6x+{24}{5}y$
$D_uf_2=\frac{174}{25}x+\frac{18}{5}y+\frac{32}{5}x+\frac{96}{25}y=>at(2,1)=>534/25$
The answer however is $\frac{774}{25}$

Your mistake was at the first derivative, you forgot to add $\frac{12}5y^2$. See this full calculation:
$$f(x,y) = x^3+5x^2y+y^3$$
$$\nabla f = \left(3x^2+10xy \atop 5x^2+3y^2 \right) $$
$$\nabla f \cdot u = \left(3x^2+10xy \atop 5x^2+3y^2 \right)\cdot \left(\frac 3 5 \atop \frac 4 5\right) =\frac {29} 5x^2+6xy+ \frac{12}5y^2 $$
$$\nabla \left(\frac {29} 5x^2+6xy+ \frac{12}5y^2\right)\cdot u = \left( \frac{58}5x+6y\atop6x+\frac{24}5 y\right) \cdot \left(\frac 3 5 \atop \frac 4 5\right) $$ $$ = \frac{174}{25}x+\frac{18}{5}y+\frac{24}5x + \frac{96}{25}y$$ $$ = \frac{294}{25}x + \frac{186}{25}y $$ Evaluating at $(2,1)$ gives: $$\left(\frac{294}{25}x + \frac{186}{25}y\right)\left(2,1\right) =\frac{588}{25}+\frac{186}{25}=\frac{775}{25}$$