This is Exercise 3.3.9 of Robinson's "A Course in the Theory of Groups (Second Edition)". According to Approach0, it is new to MSE.
The Details:
On page 6, ibid, the dihedral group $D_{2n}$ of order $2n$ is defined as the symmetry group of a regular $n$-gon.
On page 28, ibid.,
A right operator group is a triple $(G, \Omega, \alpha)$ consisting of a group $G$, a set $\Omega$ called the operator domain and a function $\alpha:G\times \Omega\to G$ such that $g\mapsto (g,\omega)\alpha$ is an endomorphism of $G$ for each $\omega\in\Omega$. We shall write $g^\omega$ for $(g,\omega)\alpha$ and speak of the $\Omega$-group if the function $\alpha$ is understood.
[. . .]
If $G$ is an $\Omega$-group, an $\Omega$-subgroup of $G$ is a subgroup $H$ which is $\Omega$-admissible, that is, such that $h^\omega\in H$ whenever $h\in H$ and $\omega\in\Omega$.
On page 60, ibid.,
[Consider] a partially ordered set $\Lambda$ with partial order $\le$. We say that $\Lambda$ satisfies the maximal condition of each nonempty subset $\Lambda_0$ contains at least one maximal element.
Analogously, the minimal condition is defined.
On page 80, ibid.,
Let $G$ be a group with operator domain $\Omega$. An $\Omega$-subgroup $H$ is called an $\Omega$-direct factor of $G$ If there exists an $\Omega$-subgroup $K$ such that $G=H\times K$; then $K$ is called an $\Omega$-direct complement of $H$ in $G$. If there are no proper nontrivial $\Omega$-direct factors of $G$, then $G$ is said to be $\Omega$-indecomposable [. . .].
On page 81, ibid
[The maximal and minimal conditions on $\Omega$-direct factors being equivalent guarantees that an] $\Omega$-group may be expressed as a direct product of finitely many nontrivial $\Omega$-indecomposable subgroups: such a direct decomposition is called a Remak decomposition.
Theorem (Remak): If an $\Omega$-group $G$ has the minimal condition on $\Omega$-direct factors, it has a Remak decomposition.
On page 92, ibid., there are the following three exercises:
Exercise 3.3.4: Prove that the central automorphisms of a group $G$ form a subgroup ${\rm Aut}_{{\rm c}}\ G$ of ${\rm Aut}\ G$.
Proof: Simply apply the one-step subgroup test.$\square$
Exercise 3.3.5 (J.E. Adney and Ti Yen): Let $G$ be a finite group which has no nontrivial abelian direct factors. Prove that $\lvert{\rm Aut}_{{\rm c}}\ G\rvert =\lvert {\rm Hom}(G_{{\rm ab}}, Z(G))\rvert$. Deduce that if $G$ has no nontrivial central automorphisms, then $Z(G)\le G'$.
Proof: This follows rather easily from the hint: "If $\alpha\in {\rm Aut}_{{\rm c}}\ G$, define $\theta_\alpha\in{\rm Hom}(G_{{\rm ab}}, Z(G))$ by $(gG')^{\theta_\alpha}=g^{-1}g^\alpha$. Show that $\alpha\mapsto \theta_\alpha$ is a bijection."$\square$
Exercise 3.3.8: Let $G=G_1\times\dots\times G_k$ be a Remak decomposition of a finite group $G$. Assume no two of the $G_i$ are isomorphic. Denote by $A$ the subgroup of all automorphisms of $G$ that leave each $G_i$ invariant. Prove the following:
(i) $A\cong{\rm Aut}(G_1)\times\dots\times{\rm Aut}(G_k)$.
(ii) ${\rm Aut}(G)=A({\rm Aut}_{{\rm c}}(G))$.
(iii) ${\rm Aut}(G)=A$ if and only if $G$ has only one Remak decomposition.
Proof sketch: (i) This follows readily from the definition of $A$.
(ii) The inclusion ${\rm Aut}(G)\supseteq A({\rm Aut}_{{\rm c}}(G))$ is clear from the definition of a set product. I'm stuck on the inclusion, ${\rm Aut}(G)\subseteq A({\rm Aut}_{{\rm c}}(G))$. I don't think it would be particularly edifying to figure out, so I've moved on.
(iii) This follows from Adney & Yen's Lemma above.$\square$
The Question:
Find the order of ${\rm Aut}(D_8\times S_3)$ (using Exercise 3.3.8).
Thoughts:
I can see this splits into three stages:
- Prove $G=D_8\times S_3$ is a Remak decomposition.
- Prove that this Remak decomposition of $G$ is unique.${}^\dagger$
- Somehow determine the order of the automorphism group of $G$ by way of, mostly, Exercise 3.3.8.
According to GAP,
gap> Size(AutomorphismGroup(DirectProduct(DihedralGroup(8),SymmetricGroup(3))));
96
gap>
the order of ${\rm Aut}(G)$ is $96.$
I have no idea how to do stage 3.
If $G$ has unique Remak decomposition${}^\dagger$
$$D_8\times S_3,$$
then ${\rm Aut}(G)\cong {\rm Aut}(D_8)\times {\rm Aut}(S_3)$.
We have $\lvert {\rm Aut}(D_8)\rvert=8$ and $\lvert{\rm Aut}(S_3)\rvert=6$. I'm not sure whether that helps.
$\dagger:$ It's not unique. (See the comments.) This follows from the GAP calculation. However, I feel like this is cheating.
Please help :)
For the proof of 3.3.8 (ii), apply the Krull-Remak-Schmidt theorem (thm 3.3.8 in Robinson) to
$$\begin{align} G &= G_1 \times \cdots \times G_k \\ &= \phi(G_1) \times \cdots \times \phi(G_k), \end{align}$$
where $\phi \in \operatorname{Aut}(G)$.
For your main question, let $G = D_8 \times S_3$.
Apply ex. 3.3.8 to conclude $\operatorname{Aut}(G) = A \operatorname{Aut}_c(G)$. Then apply ex. 3.3.8 and ex 3.3.5 to conclude $|A| = 48$, $|\operatorname{Aut}_c(G)| = 8$, and $|A \cap \operatorname{Aut}_c(G)| = 4$.
Conclude that $|\operatorname{Aut}(G)| = 96$.