In the book Elementary Differential Geometry by Andrew Pressley the following theorem is proven:
Let $U$ be an open set in $\mathbb{R}^n$ and let $f: U \to \mathbb{R}$ be smooth. Let $p \in U$ be such that $\nabla f (p) \neq 0$ (he calls this a regular point), and let $c = f(p)$. Then the set of all vectors tangent to $f^{-1}[\{c\}]$ is equal to $[\nabla f (p)]^\perp:= \{v \in \mathbb{R}^n \, | \, v \cdot \nabla f (p) = 0 \}$.
A tangent vector to $f^{-1}[\{c\}]$ at $p$ is defined as a vector $v \in \mathbb{R}^n$ such that $v = \alpha'(0)$ for some smooth $\alpha : (-1, 1) \to U$ with $\alpha(t) \in f^{-1}[\{c\}]$ for all $t \in (-1,1)$.
This theorem implies that, for all regular points $p$, the set of all tangent vectors to the level set $p$ is a vector space. In particular it is a subspace of $\mathbb{R}^n$. My question is what can we say in the case that $p$ is not a regular point.
In other words, supposing the situation is above except that $\nabla f(p) = 0$, is it true that $$\{\alpha'(0) \, | \, \alpha \in C^{\infty}((-1,1), U), \, \alpha(0) = p, \, \alpha(t) \in f^{-1}[\{c\}] \, \forall t \in (-1,1) \}$$ is a vector space?
Consider the origin in the (double) cone $x^2+y^2=z^2$. Your set of tangent vectors to curves gives you the entire cone. This is certainly not a vector space, and it spans all of $\Bbb R^3$. Relevant things to look up are tangent cone and Zariski tangent space.