I first give two definitions.
Def1: A set $S$ is a cone if $x \in S, \lambda \geq 0 \implies \lambda x \in S$.
Def2: Let $S$ be any set (we may assume $\mathbb{R}^n$ with the usual Euclidean norm) and $\bar{x} \in S$. The tangent cone to $S$ at $\bar{x}$ is defined as
$$T_S(\bar{x}) = \overline{\{ h : \bar{x} + \lambda h \in S, \text{ for some } \lambda >0\}}$$
Prove that the tangent cone is indeed a cone.
So it was a silly move in the end, and that convexity was not necessary.