I am searching for some groups, where it is not so obvious that they are groups.
In the lecture's script there are only examples like $\mathbb{Z}$ under addition and other things like that. I don't think that these examples are helpful to understand the real properties of a group, when only looking to such trivial examples. I am searching for some more exotic examples, like the power set of a set together with the symmetric difference, or an elliptic curve with its group law.
On
Here are some examples:
The ideal class group of a number field $K$. It is not obvious that this is a group, because for example to be able to invert an ideal requires the definition of an invertible ideal and showing that $\mathfrak{p}\mathfrak{p}^{-1} = \mathcal{O}$. This last part is not trivial and if memory serves me right you need to invoke the Nakayama Lemma.
The fundamental group of a topological space $X$ - not trivial to show that the operation of taking products of loops is associative. When I took such a class my lecturer drew some pretty pictures to show homotopies between $f \ast (g \ast h)$ and $ (f \ast g) \ast h$, but I was not entirely convinced.
On
Homological algebra. Let $A,B$ be abelian groups (or more generally objects of an abelian category) and consider the set of isomorphism classes of abelian groups $C$ together with an exact sequence $0 \to B \to C \to A \to 0$ (extensions of $A$ by $B$). It turns out that this set has a canonical group structure (isn't that surprising?!), namely the Baer sum, and that this group is isomorphic to $\mathrm{Ext}^1(A,B)$. This is also quite helpful to classify extensions for specific $A$ and $B$, since $\mathrm{Ext}$ has two long exact sequences. For details, see Weibel's book on homological algebra, Chapter 3. Similarily many obstructions in deformation theories are encoded in certain abelian groups.
Combinatorial game theory. A two-person game is called combinatorial if no chance is involved and the ending condition holds, so that in each case one of the two players wins. Each player has a set of possible moves, each one resulting in a new game. There is a notion of equivalent combinatorial games. It turns out that the equivalence classes of combinatorial games can be made into a (large) group. The zero game $0$ is the game where no moves are available. A move in the sum $G+H$ of two games $G,H$ is just a move in exactly one of $G$ or $H$. The inverse $-G$ of a game $G$ is the one where the possibles moves for the two players are swapped. The equation $G+(-G)=0$ requires a proof. An important subgroup is the class of impartial games, where the same moves are available for both players (or equivalently $G=-G$). This extra structure already suffices to solve many basic combinatorial games, such as Nim. In fact, one the first results in combinatorial game theory is that the (large) group of impartial combinatorial games is isomorphic to the ordinal numbers $\mathbf{On}$ with a certain group law $\oplus$, called the Nim-sum (different from the usual ordinal addition). This identification is given by the nimber. This makes it possible to reduce complicated games to simpler ones, in fact in theory to a trivial one-pile Nim game. Even the restriction to finite ordinal numbers gives an interesting group law on the set of natural numbers $\mathbb{N}$ (see Jyrki's answer). All this can be found in the fantastic book Winning Ways ... by Conway, Berlekamp, Guy, and in Conway's On Numbers and Games. A more formal introduction can be found in this paper by Schleicher, Stoll. There you also learn that (certain) combinatorial games actually constitute a (large) totally ordered field, containing the real numbers as well as the ordinal numbers. You couldn't have guessed this rich structure from their definition, right?
Algebraic topology. If $X$ is a based space, the set of homotopy classes of pointed maps $S^n \to X$ has a group structure; this is the $n$th homotopy group $\pi_n(X)$ of $X$. For $n=1$ the group structure is quite obvious, since we can compose paths and go paths backwards. But at first sight it is not obvious that we can do something like that in higher dimensions. Essentially this comes down to the cogroup structure of $S^n$. There is a nice geometric proof that $\pi_n(X)$ is abelian for $n>1$.
On
May be the groups of nimbers would fit the bill? The underlying set is that of non-negative integers $\mathbb{N}$. The group operation (denoted by $+$) is defined recursively as follows
$$
a+b=\operatorname{mex}\left(\{a'+b\mid a'<a\}\cup \{a+b'\mid b'<b\}\right).
$$
Here $\operatorname{mex}(S)$ is defined for all proper subsets of $\mathbb{N}$ and means the smallest non-negative integer not in the set $S$ (=Minimum EXcluded number). So $0+0=0$ simply because both sets on the r.h.s. are empty. But then $0+1=\operatorname{mex}(\{0\})=1=1+0$, $1+1=\operatorname{mex}(\{1\})=0$,
$0+2=\operatorname{mex}(\{0,1\})=2=2+0$, $1+2=3$, $2+2=0$ et cetera.
The operation is well defined, because the sets on the r.h.s. are obviously finite, and hence proper subsets of $\mathbb{N}$, for all $a,b\in\mathbb{N}$.
Now, it turns out that this operation is just the NIM-sum (addition in base two without carry). That is not entirely obvious even though it isn't exceedingly hard to see either.
It turns out that the sets of the form $S_n:=\{x\in\mathbb{N}\mid x<2^n\}$ are subgroups. Furthermore, if $n$ is a power of two, this set also has a multiplication that turns it into a field. The construction is due to Conway. See this wikipage for more.
On
1- Fourier analysis: the set of non-vanishing absolutely convergent Fourier series is a group under pointwise multiplication.
The neutral element is the constant function equal to $1$. And product stability follows from Cauchy product. These are straightforward. The existence of inverses is less obvious.
Wiener's lemma: if $f(t)=\sum_{\mathbb{Z}}c_ne^{int}$ is absolutely convergent, i.e. $\sum_{\mathbb{Z}}|c_n|<\infty$, and does not vanish, then $\frac{1}{f(t)}$ is also the sum of an absolutely convergent Fourier series.
This is not that hard either. But it was, and remains, striking. You can find a proof here. Less elementary, but much more interesting, the proof given by Gelfand which raised the interest in Banach algebras. Indeed, the absolutely convergent Fourier series form a unital commutative Banach algebra with spectrum $[0,2\pi]$. More precisely, the characters are the point evaluations $f\longmapsto f(t_0)$. The invertibility of non-vanishing elements is then obvious via Gelfand representation.
2- von Neumann algebras: for a type $\rm{II}_1$ factor von Neumann algebra $M$, i.e. an infinite-dimensional noncommutative probability space, we can make sense of $t\times t$ matrices over $M$ for every real $t>0$. This gives rise to another type $\rm{II}_1$ factor $M^t$.
In their seminal work dating back to the 1930's, Murray and von Neumann introduced the fundamental group of a $\rm{II}_1$ factor $M$ $$ \mathcal{F}(M):=\{t>0\;;\;M^t\simeq M\}. $$ It is not hard, but not obvious per se, to see that this is a subgroup of $(\mathbb{R}^+,\;\cdot\;)$.
One of their striking classification results says that, up to isomorphism, there exists a unique approximately finite-dimensional type $\rm{II}_1$ factor $R$. As a consequence, it follows that $$ \mathcal{F}(R)=\mathbb{R}^+. $$ By Connes' groundbreaking work (1976), any amenable type $\rm{II}_1$ factor is isomorphic to $R$. So these also have fundamental group equal to $\mathbb{R}^+$. This includes the group von Neumann algebra $L(\Gamma)$ of any countable amenable group $\Gamma$ with infinite conjugacy classes.
On the other hand, Connes proved in 1980 that the fundamental group of $L(\Gamma)$ is countable when $\Gamma$ has Kazhdan's property (T). But it remained open for some time whether the fundamental group of a $\rm{II}_1$ factor could be trivial.
In a more recent breakthrough, Popa exhibited in 2001 such examples. In particular, he showed that $$ \mathcal{F}(L(\mathbb{Z}^2\rtimes \rm{SL}(2,\mathbb{Z})))=\{1\}. $$ On the opposite side, he also proved in 2003 that any countable subgroup of $\mathbb{R}^+$ arises as the fundamental group of some type $\rm{II}_1$ factor. For a larger class of such groups and open questions, see these slides by Vaes, another important contributor to the theory.
Finally, note that Voiculescu's free probability theory allowed Radulescu to prove that $$ \mathcal{F}(L(F_\infty))=\mathbb{R}^+ $$ for the free group on a countably infinite number of generators $F_\infty$. Unfortunately, these techniques have not permitted to compute the fundamental group of $L(F_n)$ for the free group on $2\leq n<\infty$ generators. Note that the following puzzling long-standing question remains open: $$ L(F_2)\simeq L(F_3)\;? $$
On
The Brauer group of a field is not obviously a group in two ways: first it's not obvious that the group is closed under its group operation, and then it's still not obvious that inverses exist.
On
The set of exotic differentiable structures on the $n$-sphere in any given dimension is a group under the operation of connected sum, with the standard sphere being the identity element. Not at all obvious that this forms a group! For example, in dimension 7, this group is isomorphic to $\mathbf{Z}/28$.
On
What about the set of line bundles over a manifold? This forms a group, whether your line bundles are real or complex. The difference between them is also very interesting, as one is 2-torsion, and the other can be torsion-free!
Here the group operation is tensor product: showing it is a closed associative operation is fairly easy. The trivial bundle is the identity. Ah, but what are the inverses?
On
I always found the fact that braid groups are groups at all quite interesting. The elements of the group are all the different braids you can make with, say, $n$ strings. The group operation is concatenation. The identity is the untangled braid. But the fact that inverses exist is not obvious.
On
Here is an example that is a bit different, namely one that appears as a subgroup of another group, but where it is not obvious that it is a subgroup.
Let $G$ be a finite group with a proper non-trivial subgroup $H$ such that for all $g\in G\setminus H$ we have $H\cap H^g = \{1\}$ (such an $H$ is called a Frobenius complement in $G$ and if $G$ has a Frobenius complement it is called a Frobenius group).
Define $$N = \left(G\setminus\bigcup_{g\in G}H^g\right)\cup\{1\}$$
Then $N$ is a subgroup of $G$, but I am not aware of a proof of this that does not involve character theory (for a proof see for example Theorem 7.2 in Isaacs' Character Theory of Finite Groups).
($N$ is called the Frobenius kernel of $G$ and it is in fact a normal complement of $H$).
On
I was surprised to learn about the elliptic curve groups. You fix constants $a$ and $b$ take the set $S$ of points on the Riemann sphere (that is, the complex plane plus a point at infinity) of the form $$y^2 = x^3 + ax + b.$$ Then define the sum two points $p_1, p_2$ on this curve by taking the straight line through $p_1$ and $p_2$ and finding the third point $p_3 = \langle x_3, y_3\rangle$ where the line intersects $S$. Then $p_3^{-1} = \langle x_3, -y_3\rangle$ is the group sum of $p_1$ and $p_2$. It's not immediately clear that there is necessarily a point $p_3$, but there is, with suitable treatment of tangents and of the point at infinity. It's not immediately clear that the operation is associative, but it is. The point at infinity is the identity element, and the inverse of the point $\langle x, y\rangle$ is $\langle x, -y\rangle $.
On
After some study, it might become clear that the following is a group. But it seemed nonobvious the first time I saw it: the collection of all fractional linear transformations from $\mathbb{C}\cup\{\infty\}$ to itself, with formulas $$z\mapsto\frac{az+b}{cz+d}$$ such that $ad-bc\neq 0$ (in order to guarantee that you don't have a constant map), using composition as the group operation. Firstly, it takes at least a full second to believe that the composition of two such things is another such thing. Secondly, at some point you realize that any one fractional linear transformation has infinitely many representations: $z\mapsto\frac{kaz+kb}{kcz+kd}$. So a lot of your early thoughts on the topic are not 100% correct. Thirdly, associativity is no fun to confirm directly. (Again, it's not immediately obvious, but eventually you can see that this is a factor group of the $2\times 2$ general linear matrix group.)
On
It's not obvious that the collection of integers mod $p$, excluding the coset of $0$, form a group under multiplication. In particular, it's not obvious that inverses exist. You typically use the Euclidean algorithm for that.
On
If $G$ is a Lie group, with multiplication operator $m:G\times G\rightarrow G$ and inverse $i:G\rightarrow G$, then $TG$ is also a Lie group, with multiplication operator $Tm:TG\times TG\rightarrow TG$ and inverse $Ti:TG\rightarrow TG$. To see this, use the obvious notation that for $v_g\in T_g G$ and $h\in G$, $v_g\cdot h = T_g R_h(v_g)$, where $R_h:G\rightarrow G$ is right multiplication by $h$, and similarly for $h\cdot v_g$ (these operations are easily shown to be associative). Then for $g,h\in G$ and $\xi,\zeta\in \mathfrak{g}$, the Lie algebra of $G$, $$T_{(g,\,h)}m(\xi\cdot g,\,\zeta\cdot h) = \xi\cdot g\cdot h + g\cdot\zeta\cdot h = (\xi + \mathrm{Ad}_g\zeta)\cdot gh,$$ and so under the bijection between $\mathfrak{g}\times G$ and $TG$ given by $(\xi,\, g)\mapsto \xi\cdot g$, $TG$ is just $\mathfrak{g}\rtimes_{\mathrm{Ad}}G$, the semidirect product of $\mathfrak{g}$ and $G$ with respect to the adjoint action $\mathrm{Ad}$.
On
Similar to how we turn the natural numbers into the additive group of integers, and the group of integers to the multiplicative group of the rationals, let $A$ be a set with an abelian operation and an identity (abelian monoid). For $A \times A$, declare $(a_1, b_1) \tilde{} (a_2, b_2)$ iff there is a $c \in A$ such that $a_1 + b_2 +c = a_2 + b_1 + c$.
On
Another example is mentionned here:
Let $G$ be a finite group of order $n$ and $S \subset G$ be any subset. Then $$S^n = \{s_1s_2 \cdots s_n \mid s_1, s_2, \dots, s_n \in S\}$$ is a subgroup of $G$.
I was surprised the first time I saw the group of unit arithmetic functions under Dirichlet convolution. Arithmetic functions are functions $f:\mathbb{N}\rightarrow F$, where $F$ can be any field (but usually $\mathbb{C}$). The operation is $$(f\star g)(n)=\sum_{d\mid n}f(d)g\left(\frac{n}{d}\right).$$ So, here the identity is the function $$\varepsilon(n)=\left\{\begin{array}{lcl}1&:&n=1\\0&:&\text{otherwise}\end{array}\right.$$while inverses are defined recursively, as described here under "Dirichlet inverse." Note that $1/f(1)$ appears in the definition of the inverses, so we must include only arithmetic functions for which $f(1)$ is invertible in $F$ (this is why we say unit arithmetic functions).