A parametric curve $C$ can be defined as follows
$$ C(p) = \{x(p), y(p) \}, \; p \in [0, 1] $$
where $p$ is the parameter.
We can define the unnormalised tangent to the point of the curve corresponding to $p$ as follows
$$ \frac{d C}{d p} = \left[\frac{d x}{d p}, \frac{d y}{d p}\right] $$
Why is that? What's the meaning of the derivative of a curve? It is apparently the tangent, but mathematically why is that? Is $\frac{d C}{d p}$ considered a gradient?
Let's write the derivative of a curve at point $p_0$ as follows: \begin{equation} \begin{split} \frac{dC}{dp}(p_0) &= \lim_{h \to 0} \left[ \frac{x(p_0+h) - x(p_0)}{h}, \frac{y(p_0+h) - y(p_0)}{h}\right] = \\ &= \lim_{h \to 0} \frac{1}{h} \left[ x(p_0+h) - x(p_0), y(p_0+h) - y(p_0)\right] = \\ &= \lim_{h \to 0} \frac{1}{h} \left( C(p_0 + h) - C(p_0) \right) \end{split} \end{equation} From the expression above, you can see that the derivative of the curve $C(p)$ can be considered as a difference between two points on a curve: $C(p_0 + h)$ and $C(p_0)$. This difference defines a vector. As $h \to 0$ this vector becomes shorter, however multiplying by the factor $\frac{1}{h}$ "rescales" it back. Therefore, tangent vector is defined as the vector between infinitely close points on a curve.
Hope that helps a bit.