I'm trying to understand the concept of the semidirect product of two groups.
An application is this answer to a question, where for $p>q$ primes, $q \mid p-1$, there is a non-abelian group of order $pq$ isomorphic to
$$C_p\rtimes_{\phi} C_q$$
for some homomorphism $\phi: C_q\to\mathrm{Aut}(C_p)\cong C_{p-1}.$
I'm trying to understand what this homomorphism looks like.
The answer states the following at point 2:
(Note the author defines $P=\langle x\mid x^p=1\rangle$ and $Q=\langle y\mid y^q=1\rangle$.)
Now, since $Q=\langle y\rangle$ normalizes $P=\langle x\rangle$, the map $\phi_k:P\to P$ given by $\phi_k(x)=y^kxy^{-k}$ is well defined. Moreover, it is clearly an automorphism with inverse $\phi_{-k}$. Finally, since $\phi_{k}\phi_j=\phi_{k+j}$, the map $y^k\mapsto\phi_k$ defines a homomorphism $\phi:Q\to \mathrm{Aut}(P)$.
Here's where I'm a bit confused - something I'm doing from here on must be wrong, but I'm not sure what. Let's represent $C_p$ and $C_q$ as addition modulo $p$ and $q$, thus the $pq$ elements of $C_p\rtimes_{\phi} C_q$ will be the tuples of the form $(a, b)$ for $a =0, ..., p-1$, $b=0, ..., q-1$.
In the equation $$\phi_k(x)=y^kxy^{-k}$$
$y \in C_q$ so it'll be of the form $(0, m)$ in the semidirect product, $x \in C_p$ will be of the form $(n, 0)$. Thus $y^kxy^{-k}=(0, mk) \circ (n, 0) \circ(0,-mk)$ where $\circ$ is the group operation of the semidirect product.
First, I thought this was just like addition on the tuples, and so I will always get $(n, 0)=x$. This interpretation must be wrong, since then $\phi_k$ would be the trivial homomorphism, and we just get the abelian direct product.
But $\circ$ is defined as $(n_1, h_1) \circ (n_2, h_2) = (n_1 \phi_{h_1}(n_2), h_1 h_2)$, which uses what will be $\phi_k$ in its own definition... so what's going wrong? I don't get how I can actually calculate binary operations on the group.
One thing I see wrong is that despite choosing additive notation for $C_p$ and $C_q$, you then use multiplication, for example in the expression $h_1 h_2$. But this is a minor error. I suggest dumping additive notation, which you are free to do. Since $C_p$ and $C_q$ are being used as subgroups of a semidirect product which is not going to be abelian, stick with the multiplicative notation $C_p=\langle x \rangle$ and $C_q = \langle y \rangle$. Also, I'm going to ignore the $P,Q$ notation and other doubled up notations. Why confuse yourself by doubling up on notations when one single notation will do?
Regarding $\phi_k$, the expression $\phi_k(y) = y^k x y^{-k}$ is internal to the semidirect product. As you say, this does not make sense in giving an external definition for a homomorphism $C_q \mapsto \text{Aut}(C_p)$.
So, let's just give an external definition of a homomorphism $\Phi : C_q \to \text{Aut}(C_p)$.
Start with the fact that the generator $y \in C_q$ has order $q$. Our job is to choose $\Phi(y) \in \text{Aut}(C_p)$ to be a nontrivial automorphism of the group $C_p$ such that the order of $\Phi(y)$ in the group $\text{Aut}(C_p)$ is equal to $q$ (making the choice so that the order divides $q$ would be sufficient; but we can actually make it equal to $q$). Once that job is done, the formula $$\Phi(y^k) = \underbrace{\Phi(y) \circ \ldots \circ \Phi(y)}_{\text{$k$ times}} $$ becomes a well-defined homomorphism, using the composition operation $\circ$ which is the group operation on $\text{Aut}(C_p)$.
The fact that one may choose $\Phi(y)$ in this manner follows by combining the hypothesis $q \mid p-1$ with the theorem that $\text{Aut}(C_p)$ is a cyclic group of order $p-1$.
Now, you stated that you want to know what the homomorphism $C_q \mapsto \text{Aut}(C_p)$ looks like. So it's possible that this answer so far is not enough for you: perhaps you really want to know what an order $q$ element of the group $\text{Aut}(C_p)$ looks like. For that purpose, I would say that you should go read the proof of that theorem.