Let $Q:=\begin{pmatrix}1&0\\0&-1\end{pmatrix}$ and $G:=\{A\in GL_2(\Bbb{R}): A^tQA=Q\}$. I want to find the Lie algebra of $G$.
I know that by definition $Lie(G):=\{X: \exp(tX)\in G ~~\forall t\in \Bbb{R}\}$. And I also know that one has the characterization that $Lie(G)=T_I(G)$ which is the tangent space of $G$ at identity $I$. I somehow don't know which works better to find $Lie(G)$.
I know that if I take $A=\begin{pmatrix} a&b\\c&d\end{pmatrix}$ then $A^tQA=\begin{pmatrix} a^2-c^2&ab-cd\\ab-cd&b^2-d^2\end{pmatrix}\stackrel{!}{=}\begin{pmatrix}1&0\\0&-1\end{pmatrix}$. But I don't know if this helps me further. I thought about If I can find a basis of $G$ and then derive it, I will get a basis of the tangent space and thus one of the Lie algebra.
Can someone help me further?
You can find it using your definition.
Let $X\in \mathrm{Lie}(G)$. Then, by definition, $e^{\lambda X}\in G$ $\ \forall \lambda\in\mathbb{R}$.
So it holds that $e^{\lambda X^t}Qe^{\lambda X}=Q$. Since $G$ is closed in $GL_2(\mathbb{R})$ we can take derivative with respect to $\lambda$ on both sides and evaluate at $\lambda=0$. This yields $X^tQ=-QX$. Hence, we just showed that $\mathrm{Lie}(G)\subseteq S:=\{X\in M_2(\mathbb{R})\mid X^tQ+QX=0\}$. In particular $S$ is the set of all $2\times2$ matrices of the form $\begin{bmatrix}0&a \\ a&0\end{bmatrix}$ for $a\in\mathbb{R}$.
Now we claim that $S\subseteq \mathrm{Lie}(G)$. So, let $X\in S$. Then $X^tQ+QX=0\Rightarrow X^t=-QXQ^{-1}$. Notice that $\forall\lambda\in\mathbb{R}$: $$e^{\lambda X^t}Q=e^{-\lambda QXQ^{-1}}Q=Qe^{-\lambda X}Q^{-1}Q=Qe^{-\lambda X} \Rightarrow e^{\lambda X^t}Qe^{-\lambda X}=Q\Rightarrow e^{\lambda X}\in G \Rightarrow X\in \mathrm{Lie}(G).$$
Alternatively, a direct computation, given that $e^{\lambda X^t}=e^{\lambda X}=\begin{bmatrix}\cosh(\lambda a)&-\sinh(\lambda a)\\\sinh(\lambda a)&\cosh(\lambda a)\end{bmatrix}$, gives $e^{\lambda X^t}Qe^{\lambda X}=Q$ for $a\not=0$ (if $a=0$ we get the undetermined case $Q=Q$).
We conclude that $\mathrm{Lie}(G)=S=\{X\in M_2(\mathbb{R})\mid X^tQ+QX=0\}$.