I am a not experienced in linear algebra, and I am not really sure how to tackle this problem. Thanks in advanced.
Show that,
$$\nabla u \cdot \mathbf m=(\mathbf m \cdot \mathbf n)\nabla u \cdot\mathbf n + (\mathbf m \cdot \mathbf t)\nabla u \cdot \mathbf t$$
where,
$$\mathbf m=(An_{x} +Bn_{y})\mathbf i + (Bn_{y} +Cn_{y})\mathbf j$$
is the vector in the direction of the conormal to the boundary.
The formula for $\mathbf m$ is irrelevant. The fact that $\mathbf t,\mathbf n$ forms an orthonormal basis for $\Bbb R^2$ tells you that for any vector $\mathbf m$ we have $$\mathbf m = (\mathbf m\cdot \mathbf t) \mathbf t + (\mathbf m\cdot\mathbf n)\mathbf n.$$ Now dot with $\nabla u$.