I'm looking to represent the automorphism group of a finite group in a small group theory package I'm writing.
I was thinking of representing a generic automorphism $a \in \operatorname{Aut}(G)$ as a pair $(o,i)$ representing the composition of an outer automorphism $o \in \operatorname{Out}(G)$ by an inner automorphism $i \in \operatorname{Inn}(G)$. Thus I can write $i$ as an element of $G / \operatorname{Center}(G)$, and use a set of coset representatives for $\operatorname{Out}(G)$. With this information, I can at least enumerate the elements of $\operatorname{Aut}(G)$.
Now, I'd like eventually to use the group structure on $\operatorname{Aut}(G)$. I guess that I'd need to write $\operatorname{Aut}(G)$ as a semidirect product to be able to compute the composition of elements of $\operatorname{Aut}(G)$ as represented by those pairs. Now, as some automorphism groups do not split in that way, I guess I'm out of luck?
I'm not so familiar with the group extension problem.
Yes, you are indeed out of luck when that extension is nonsplit! Representing ${\rm Aut}(G)$ for finite groups $G$ is one of the more difficult problems in computational group theory.
The default approach is to look for as small a subset of $G$ as you can find (unions of conjugacy classes are a good thing to try) on which ${\rm Aut}(G)$ acts faithfully, and to use that to get a permutation representation of ${\rm Aut}(G)$, but in difficult examples that can lead to very large degrees.
For finite simple groups $G$, it is useful to store suitable representation of ${\rm Aut}(G)$ - so for example, for $A_n$ with $n \ne 6$ you can use $S_n$.
Also, lots of finite solvable groups have solvable automorphism groups, and you can represent those using power-conjugate representations.