Hi suppose I have a functional $I_{x,y}$ acting on continuous paths
$$\{\xi~:~\xi:[0,1]\to \mathbb{R}^n,~\xi(0)=x,~\xi(1)=y\}$$ .
In my case $I_{x,y}[\xi]=\int_0^1 L(\xi(t),\dot{\xi}(t))dt$ where $L(\xi(t),\dot{\xi}(t))\in \mathbb{R}$ is some function.
How can I write
$$ \partial_\epsilon I_{x,y}[\xi+\epsilon \eta] $$
in terms of
$$ \nabla_y I(x,y)\cdot \eta ~~~?~\text{or something like this }. $$
I think this has something to do with directional or functional derivative but im very confused.
The mapping from $\eta$ to the directional derivative that you are asking to evaluate is exactly what we call the functional derivative of $I_{x,y}$. For notation let's say the first argument of $L$ is denoted by $q$ and the second is denoted by $p$ then if you replace $\xi$ by $\xi + \epsilon \eta$ then the argument of $L$ becomes $L(\xi+\epsilon \eta,\dot{\xi}+\epsilon \dot{\eta})$. Thus linearization of $L$ (which is just an ordinary scalar function of two scalar variables) yields $L(\xi,\dot{\xi})+L_q(\xi,\dot{\xi}) \epsilon \eta + L_p(\xi,\dot{\xi}) \epsilon \dot{\eta}$, using the ordinary multivariable chain rule (nothing functional-analytic going on here). Thus the functional derivative of your $I_{x,y}$ is the mapping
$$\xi \mapsto \left ( \eta \mapsto \left ( \int_0^1 L_q(\xi(t),\dot{\xi}(t)) \eta(t) + L_p(\xi(t),\dot{\xi}(t)) \dot{\eta}(t) dt \right ) \right )$$
Now if you are looking for extrema of $I_{x,y}$, as you probably are, this is not entirely useful because not all $\eta$ are admissible with your boundary condition. Specifically if you want to find a critical point of $I_{x,y}$ subject to the boundary condition, only $\eta$ that vanish on the boundaries are admissible. This can be incorporated into this formalism by integration by parts:
$$\int_0^1 L_p(\xi(t),\dot{\xi}(t)) \dot{\eta}(t) dt = -\int_0^1 \frac{d}{dt} \left [ L_p(\xi(t),\dot{\xi}(t)) \right ] \eta(t) dt$$
where the boundary term has vanished because $\eta(0)=\eta(1)=0$.
Now your functional derivative at a fixed point takes the form $\eta \mapsto \int_0^1 F[\xi(t),\dot{\xi}(t)] \eta(t) dt$. The $\eta$ here are "sufficiently general" that, for this to vanish for all $\eta$, one must have the differential equation
$$\frac{d}{dt} \left [ L_p(\xi(t),\dot{\xi}(t)) \right ] = L_q(\xi(t),\dot{\xi}(t)).$$
This is called the Euler-Lagrange equation. This very last step takes some theoretical machinery to prove, but the rest of the derivation is just ordinary calculus.