The map: $T:\mathbb{M}_{+}^3 \rightarrow \mathbb{S}^3$ is given to be continuous, where $T(\boldsymbol{F})= \boldsymbol{F}^{\mathrm{T}} \boldsymbol{F}.$
I was trying to verify this using the basic $\epsilon-\delta$ definition of continuity, using the idea that the norm would be the $\sqrt{tr(A^TA)}$, but I was ending up stuck. How do I approach this problem? In general, I am struggling to prove continuity for maps on spaces of matrices.
Let us assume that the field of coefficients for your matrices is contained in the real numbers otherwise, square roots get tricky. A road map on how to do this:
Lets examine the expression $\sqrt{ \text{tr}(A^TA)}$. The trace is just the sum of the diagonal entries. The $(i,i)$th entry of $A^TA$ is just the sum of squares of the elements in the $i$-th column. Therefore if we write $A=(a_{ij})$, we have that $$\sqrt{ \text{tr}(A^TA)} = \sqrt{\sum_{i,j}a_{ij}^2}$$
This is the same metric that we get when we embed your matrix spaces in $\mathbb R^9$ coordinate-by-coordinate and give them the inherited Euclidean metric.
Then prove that a function $f: \mathbb R^m \to \mathbb R^n$ is continuous if and only if the component functions s are continuous. That is, write $f = (f_1(\mathbf{x}),\ldots,f_n(\mathbf{x}))$ where $f_i$ are functions $\mathbb R^m \to \mathbb R$.
Prove that polynomial functions $p:\mathbb R^n \to \mathbb R$ are continuous.
Notice that $F^TF$ is just an $m$-tuple of polynomial functions each from $\mathbb R^9 \to \mathbb R$.
Steps 3 and 4 are where you'll get to do your $\epsilon-\delta$ proofs.