Let $\text{I}(\mathbb{R}^n)$ be the set of all isometries of $\mathbb{R}^n$, and let $\text{O}(n)$ be the orthogonal group consists of $n \times n$ matrices.
And, I would like to show that $\text{I}(\mathbb{R}^n)$ is not isomorphic to $\text{O}(n) \times \mathbb{R}^n$.
Hint:
Actually, my textbook suggests to find some $f \in \text{O}(n)$ and $g \in \mathbb{R}^n$ such that $fg \neq gf$.
That being said, it looks like we have to say that the group $\text{I}(\mathbb{R}^n)$ is not a normal subgroup of the other one.
Questions:
But, I think $gf$ is not even defined because $g$ is a $n \times 1$ matrix (column vector) while $f$ is a $n \times n$ matrix.
Also, can we not define a map $\text{M}: \text{I}(\mathbb{R}^n) \to \text{O}(n) \times \mathbb{R}^n$ such that $\text{M}(I)=(A,a)$ where $I(x) = Ax + a$ because $I$ is an affine map.
So, I wonder why the hint is given like this. Moreover, what is the consequence when $\text{I}(\mathbb{R}^n)$ is not a normal subgroup of $\text{O}(n) \times \mathbb{R}^n$?
To your first question:
By $f$ and $g$ they really mean the maps $x\mapsto fx$ and $x\mapsto x+g$. Technically speaking, these maps are the images of $f$ and $g$ under the inclusions $O(n)\to I(n),A\mapsto(x\mapsto Ax)$ and $\mathbb R^n\to I(n),v\mapsto(x\mapsto x+v)$.
To your second question:
We can certainly define such a map. But will it be a homomorphism? Composing $Ax+a$ with $Bx+b$ we get $BAx+Ba+b$, which would be mapped to $(BA,Ba+b)$, while composing $(A,a)$ and $(B,b)$ gives $(BA,a+b)$.
To your third question:
In $I(n)$, rotations don't commute with translations. In $O(n)\times\mathbb R^n$ they do. So the two groups can't be isomorphic. As for the consequences of one not being a normal subgroup of the other: I don't know what you're looking for here. They're not just not normal subgroups of each other, they aren't subgroups of each other at all. The main takeaway is that rotations and translations don't act independently of each other, if this helps you?