Let $G$ be the group defined by $$G = \{A \in \mathrm{GL}(2,\Bbb R) \mid A^tA=r^2I, r>0,\det A >0\}.$$ Find the Lie algebra of $G$.
I've managed to figure out that the elements of $G$ take the form $$A=\begin{pmatrix}a&-b\\ b&a\end{pmatrix}, \ a^2+b^2=r^2$$ by some elementary trigonometry.
I'm having difficulties figuring out what $T_IG$ should be. If $X \in T_IG$ and $c:(a,b) \to G$ is a smooth path with $c(0)=I$ and $c'(0)=X$, then $c(t) \in G$ and we have two conditions $$c(t)^Tc(t)=r^2I, \ \det c(t)>0.$$
From $\det c(t) >0$ we get that $$\operatorname{tr} X >0$$ so the elements of the diagonal on $X$ should have a positive sum. Can I apply the product rule to $c(t)^Tc(t)=r^2I$? That is $$(c(t)^T)'c(t)+c(t)^Tc'(t)=0?$$ This would give that $X^T+X=0$ i.e. that $X$ is skew-symmetric?
I think you have misunderstood the definition of $G$: $r$ is not a fixed value, but a free parameter (indeed, if $r$ was fixed, it could only be $1$ since $I_2$ must belong to $G$, as a subgroup of $GL_2(\Bbb R)$). From your computation, it appears that $$ G = \left\{\begin{pmatrix} r\cos \theta & -r\sin \theta \\ r\sin \theta & r \cos \theta \end{pmatrix} \mid r \neq 0, \theta \in \Bbb R \right\}. $$ The Lie algebra of $G$ is given by the tangent space at $I_2$. Let $$c(t) = \begin{pmatrix} r(t)\cos \theta(t) & -r(t)\sin \theta(t) \\ r(t)\sin \theta(t) & r(t) \cos \theta(t) \end{pmatrix} = r(t)\begin{pmatrix} \cos \theta(t) & -\sin \theta(t) \\ \sin \theta(t) & \cos \theta(t) \end{pmatrix}$$ be a curve with $c(0) = I_2$, i.e $r(0) = 1$ and $\theta(0) = 0 \pmod 2\pi$. Leibniz rule yields $$ c'(t) = r'(t)\begin{pmatrix} \cos \theta(t) & -\sin \theta(t) \\ \sin \theta(t) & \cos \theta(t) \end{pmatrix} + r(t) \theta'(t) \begin{pmatrix} -\sin \theta(t) & -\cos \theta(t) \\ \cos \theta(t) & -\sin \theta(t) \end{pmatrix} $$ and at $t=0$, this yields $$ c'(0) = r'(0) I_2 + \theta'(0)\begin{pmatrix}0 & -1 \\ 1 & 0 \end{pmatrix}. $$ One then sees that all these tangent vectors at $I_2$ have the form $$ \begin{pmatrix} a & -b \\ b &a \end{pmatrix} $$ for some $a,b \in \Bbb R$, without any further restriction.