I have a question about the Euler totient function. I am new to the number theory and I don't know where to start to prove this.
If $p$ is a prime, show that $\phi(\phi(p^k)) = p^{k-2}\phi((p-1)^2)$ for integers $k \ge 2$, where $\phi (n)$ is the Euler totient function.
We can prove the proposition using астон вілла олоф мэллбэрг's hint.
We'll need the property that $$\phi(mn) = \phi(m)\phi(n) \tag{1}$$ for coprime $m$ and $n$, and the fact that $$\phi\big(p^k\big) = p^k - p^{k - 1} = p^{k - 1}(p - 1) \tag{2}$$ for prime $p$ and nonnegative integral $k$.
Combining Properties $(1)$ and $(2)$, we get the general formula of $\phi(n)$ for any arbitrary $n$: $$\phi(n) = n \prod\limits_{\substack{p \,\mid\, n \\ p \text{ prime}}} \left(1-\frac{1}{p}\right) \tag{3}$$
(As @ccorn pointed out, the formula in the question is actually correct, contrary to what Milten said in the comments. We will need Property $(3)$ to prove that the formula we derive is equal to the given equation.)
I won't prove the first property here, but we can easily prove the second one.
By definition, $\phi\big(p^k\big)$ is equal to the number of coprime integers between $1$ and $p^k$. But the only prime factor of $p^k$ is $p$, which means it is divisible by every multiple of $p$ less than $p^k$. This means it is divisible by $1p, 2p, 3p, \dots,(p^{k - 1})p$. As we can see, the coefficients conveniently number the non-coprime integers for us (i.e., $1p$ is the first non-coprime number, $2p$, is the second non-coprime number, and so on.)
Therefore, $\phi\big(p^k\big) = p^k - p^{k - 1}$. (The logic being, $\phi\big(p^k\big)$ is equal to the number of integers between $1$ and $p^k$, which is simply $p^k$, minus the number of non-coprime integers, which we calculated to be $p^{k - 1}$.)
Armed with these two properties, we can prove the given proposition. $$\phi\left(\phi\big(p^k\big)\right) = \phi(p^{k - 1}(p - 1)) = \phi(p - 1)\phi\big(p^{k - 1}\big)$$ since $p^{k - 1}$ and $p - 1$ are coprime.
Applying Property $(2)$ again, we get that $\phi\big(p^{k - 1}\big) = p^{k - 2}(p - 1)$.
Hence, $$\phi\left(\phi\big(p^k\big)\right) = p^{k - 2}(p - 1)\phi(p - 1) \tag{4}$$
(Notice that we cannot simplify $\phi(p - 1)$ further because $p - 1$ is not prime for all $p$. All primes with the exception of $2$ are odd, which means $p - 1$ is divisible by $2$, making them composite. And for $p = 2$, $p - 1 = 1$, which is considered to be neither prime nor composite.)
Using Property $(3)$, we can prove that $\phi\big(m^n\big) = m^{n - k}\phi\big(m^k\big)$ for any integral $k$ such that $1 \le k \le n$. We know that $$\begin{align*} \phi\big(m^n\big) &= m^n \prod\limits_{\substack{p \,\mid\, m \\ p \text{ prime}}} \left(1-\frac{1}{p}\right) \\\\ &= m^{n - k} \left(m^k \prod\limits_{\substack{p \,\mid\, m \\ p \text{ prime}}} \left(1-\frac{1}{p}\right) \right) \\\\ &= m^{n - k}\phi\big(m^k\big) & \text{(By Property $(3)$)} \end{align*}$$
By applying the result we just proved, we can write Equation $(4)$ as $$\phi\left(\phi\big(p^k\big)\right) = p^{k - 2}\phi\big((p - 1)^2\big) \tag*{$\blacksquare$}$$