I would like to ask - why does $\operatorname{Out}(G)$ := ${\operatorname{Aut}(G) / \operatorname{Inn}(G)}$ make sense? As $\operatorname{Out}(G)$ is outer automorphism group, I would expect it to include outer automorphisms, but for nontrivial $\operatorname{Inn}(G)$ the quotient is a group of cosets which are sets of functions, not functions. Isomorphism would be totally intuitive for me but equality I do not understand.
Thank you in advance!
On
The set of outer automorphisms (taking an "outer automorphism" of a group to be an automorphism that is not inner) does not give a group. The easiest way to see this is to consider that the identity automorphism is an inner automorphism. But there can also be other cases where the composition of two outer automorphisms gives a nontrivial inner automorphism.
Thus it makes sense to consider equivalence classes of outer automorphisms. Looking at $\mathrm{Aut}(G)/\mathrm{Inn}(G)$, you have 1: an actual group, and 2: every nonzero equivalence class consists entirely of outer automorphisms.
On
I want to just make a quick addition to the other answers:
Some people will conflate “outer automorphism” and “automorphism which is not inner” in casual math conversations. This may be a source of confusion — these are distinct ideas.
Although outer automorphisms are not actual functions on $G$, they do induce functions on the conjugacy classes of $G$. (Why?) You might hear talk of outer automorphisms acting like functions — this is one context in which such language occurs.
So indeed one defines $Out (G) := Aut(G)/Int(G)$. But an element of $Out(G)$, an outer automorphism, is not an automorphism of $G$ (as you notice if I understood correctly) - the naming is confusing. It is just what it is, an automorphism up to composing with an inner automorphism.
By the way, there is no definition of when an automorphism of $G$ is an outer automorphism.