To start with, I'm a physicist and I try to understand group theory for physics (also I am back in physics after two years so I forgot some things).
I'm learning from an online course, and I can't understand how to get the irreducible representations for $O(2)$.
First, I started with $SO(2)$, $$R(\theta)=\begin{pmatrix} \cos\theta & -\sin\theta\\ \sin\theta & \cos\theta \end{pmatrix}$$
And I understood that in order to find the representations, I use a similarity transformation and I find $$ D(\theta) = \begin{pmatrix} e^{-i\theta} & 0\\ 0 & e^{i\theta} \end{pmatrix}$$
Then I understand that the irreps are given by: $ D^{(1)}(\theta) = e^{-i\theta} $ and $ D^{(-1)}(\theta) = e^{i\theta} $. Hope that what I said is correct.
Now, about the $O(2)$ is made of SO(2) matrices, plus de matrices $$R^{'}(\theta) = \begin{pmatrix} \cos\theta & \sin\theta\\ \sin\theta & -\cos\theta \end{pmatrix}$$
I understand that the new representations are: previous $D(\theta)$ and $$D^{'}(\theta) = \begin{pmatrix} 0 & e^{-im\theta}\\ e^{im\theta} & 0 \end{pmatrix}$$
Can someone explain me how to get the $D^{'}(\theta) $ matrix and how can I find the irreps of O(2)?
Following the discussion in the comments, I will ignore the word "defining" and demonstrate how to classify the irreducible representations of $O(2)$. I will also not use the words "induced representation", as you might not have come across those yet.
First, a correction:
This is not right. The correct statement is that the irreducible complex representations are all one-dimensional and have the form $D^{(n)}(\theta) = e^{in\theta}$ for any integer $n$. If you don't understand this then you should review your notes.
Now to $O(2)$. Let $\tau \in O(2)$ be a reflection, so that $O(2) = SO(2) \rtimes \langle \tau \rangle$. Let $V$ be an irreducible complex representation of $O(2)$. Then in particular $V$ is a (usually not irreducible) complex representation of $SO(2)$. Let $U \leq V$ be an irreducible $SO(2)$-submodule. Then $U$ is one-dimensional and $R_\theta \in SO(2)$ acts as $e^{in\theta}$ for some integer $n$. Now consider the subspace $\tau U \leq V$. We calculate that $$R_\theta \tau u = \tau R_{-\theta} u = e^{-in\theta} \tau u \qquad (v \in U).$$ Hence $\tau U$ is also an $SO(2)$-submodule and $U + \tau U$ is an $O(2)$-submodule. Since $V$ is irreducible, $V = U + \tau U$. If $U = \tau U$ then we must have $n = 0$ and $V$ is a one-dimensional representation of $\langle \tau \rangle$, so $\tau$ acts either as $1$ (the trivial representation) or $-1$. Otherwise, $V$ is two-dimensional. If $u \in U \setminus 0$ then with respect to the basis $u, \tau u$ we have that $R_\theta$ acts as $$\begin{pmatrix} e^{in\theta} & 0 \\ 0 & e^{-in\theta} \end{pmatrix}$$ and $R_\theta \tau$ acts as $$\begin{pmatrix} 0 & e^{in\theta} \\ e^{-in\theta} & 0 \end{pmatrix}.$$ Hope that helps you see where those matrices come from.