Given:
\begin{equation*} f(x,y) = e^{xy}, \quad P=(1,1),\ \mathbf{v} = (-\frac{\sqrt{3}}{2}, \frac{1}{2}) \end{equation*}
I am trying to calculate the directional derivative of $f$ in direction of $\mathbf{v}$.
My problem is, that when I apply two different ways of caculating the derivative I get different results. However I do not find a mistake in my calculation:
1) First approach:
\begin{align*} f(P) &= e^{1*1} = e \\ P + t \mathbf{v} &= (1,1)^T + (-\frac{t\sqrt{3}}{2}, \frac{t}{2})^T = (1 -\frac{t\sqrt{3}}{2}, 1 + \frac{t}{2})^T \\ \end{align*}
\begin{equation*} (1 -\frac{t\sqrt{3}}{2})(1 + \frac{t}{2}) = 1 + \frac{t}{2} - \frac{t \sqrt{3}}{2} - \frac{t^2 \sqrt{3}}{4} \end{equation*} \begin{align*} f(P + t \mathbf{v}) = e^{1 + \frac{t}{2} - \frac{t \sqrt{3}}{2} - \frac{t^2 \sqrt{3}}{4} } \\ \lim_{t \rightarrow 0} \frac{f(P + t \mathbf{v}) - f(P)}{t} = \lim_{t \rightarrow 0 } \frac{e^{1 + \frac{t}{2} - \frac{t \sqrt{3}}{2} - \frac{t^2 \sqrt{3}}{4} } - e}{t} = \lim_{t \rightarrow 0 } \frac{0}{t} = 0 \end{align*}
2) Second approach: The directional derivative should be equal to the Jacobian of $f$ at $P$ times $\mathbf{v}$:
\begin{align*} J(f) &= [ye^{xy}, xe^{xy}] \\ J(f, P) &= [e, e] \\ J(f, P) \mathbf{v} &= [e, e] (-\frac{\sqrt{3}}{2}, \frac{1}{2})^T \\ &= \frac{e}{2}(1 - \sqrt{3}) \end{align*}
However that is not equal to zero. I am inclined to believe that there is a mistake in my first approach, but the calculations seem correct.
First notice that you've made a mistake here:
If you distribute the $t$ in the last equation you get $1 + t/2 -t(\sqrt 3/4)-t^2(\sqrt 3/4)$ regardless I don't see the point of your last equality since $1 + \frac{t}{2} - \frac{t \sqrt{3}}{2} - \frac{t^2 \sqrt{3}}{4}$ is perfectly fine to work with when taking a limit as $t \to 0.$
Your biggest mistake however comes from when you calculate the limit as $t \to 0$ of $$Q_t = \frac{f(P+tv)-f(P)}{t} = \frac{\exp(1+ \frac t 2 - \frac{\sqrt3}{2}t - t^2 \frac{\sqrt 3}{4} ) - e}{t}.$$
It is correct that the numerator converges to $e- e = 0$ but this doesn't mean that $Q_t \to 0$ since the numerator also converges to $0$.
You can use l'hospital's rule to find this limit. Taking the derivative of the numerator and denominator you get
$$ \frac{(1/2 - \sqrt 3/2 - t(\sqrt 3/2) )\exp(1+ \frac t 2 - \frac{\sqrt3}{2}t - t^2 \frac{\sqrt 3}{4})}{1}$$
which converges to $$\left(\frac{1}{2} - \frac{\sqrt 3}{2} \right) e = \frac{e}{2}(1 - \sqrt 3).$$