Let $S:\mathbb{R}^2 \rightarrow \mathbb{R}^2$ given by rotation counterclockwise around the origin by an angle of $\pi/4$, and let $T:\mathbb{R}^2 \rightarrow \mathbb{R}^2$ given by reflection over the $x-$axis.
Q.1) Write the matrix form of $U:=S\circ T$.
Q.2) For each integer $n>0$ let $\vec{v}_{n} =(0, \frac{1}{n})$. Compute the limit: $$\lim_{n\to \infty} U(\vec{v}_{n})$$ or prove it doesn't exist.
1) I think that the matrix form of $S$ is:
\begin{bmatrix}\sin \pi/4 &-\cos \pi/4 \\\cos \pi/4&\sin \pi/4\end{bmatrix} and the matrix form of $T$ is: \begin{bmatrix}1 &-1 \\ 0 & 0\end{bmatrix}.
Is this correct? If so, is $S \circ T$ simply equal to the product of two matrices?
What'd be a general strategy for solving 2)?
The $S$ matrix is not correct. It should be, thinking in counterclockwise rotations with an angle $\theta$
See this for a derivation of this matrix that is animated.
The matrix $T$ is also not correct. $T$ must reflect over the $x$-axis so, for example, it should make $(-1,1) \to (-1,-1)$. But your $T$ makes
$$T\left(\begin{matrix}-1\\1\end{matrix}\right) = \left(\begin{matrix}-2\\0\end{matrix}\right) $$
The correct transformation makes $x$ be the same and changes the sign of the $y$ part.
$$\tilde{T}\left(\begin{matrix}x\\y\end{matrix}\right) = \left(\begin{matrix}x\\-y\end{matrix}\right) $$
Define $\tilde{T} := \left(\begin{matrix}a& b\\c& d\end{matrix}\right)$ then you'll have that
$$\left \{\begin{matrix}ax+by=x\\cx+dy=-y\end{matrix}\right.$$
Solving you'll get that $a=1,b=0,c=0,d=-1$ and the correct matrix should be
See that, let $\theta = \pi/4$
$$U = S\tilde{T} = \left(\begin{matrix}\cos(\pi/4)& \sin(\pi/4)\\\sin(\pi/4 ) & -\cos(\pi/4)\end{matrix}\right)$$
So, yes, the composition will be the product of the matrices and also we can compute the $U(v_n)$. It will be just
$$U(v_n) = \left(\begin{matrix}\frac{\sin(\pi/4)}{n}\\\frac{-\cos(\pi/4)}{n}\end{matrix}\right) $$
Note that $U$ is constant, therefore continuous and the limit can get inside the application. So we could just see the limit of $v_n$.
$$\lim_{n \to \infty}U(v_n) \stackrel{U\text{ is continuous}}= U\left(\lim_{n \to \infty}v_n\right) = U\left(\left(\begin{matrix}0\\0\end{matrix}\right) \right)$$