I found this algorithm (Algorithm 4.79 in this document) However, I am unable to understand what the run-time complexity of the algorithm might be.
My first guess is that the for loop is bounded by $O(\log k)$ (number of prime factors of the group order) but I am not able to see what the complexity of the inner while loop might be bounded by. Can anyone help me with this?
Here is the link to the algorithm 4.79: https://cacr.uwaterloo.ca/hac/about/chap4.pdf
One input is $$n=\#G = \prod_{k=1}^{\omega(n)} p_k^{e_k}$$
The algorithm then works out the exponents of $p_k$ in the order of $\alpha$ by probing whether $\alpha^{\textstyle n/p_k^{q_k}} \stackrel?= 1$ where $0\leqslant q_k \leqslant e_k$. In the worst case, all values from 0 to to $e_k$ have to be checked for all primes that divide $\#G$. This means that the inner 2 loops perform at most
$$\sum_{k=1}^{\omega(n)} e_k = \Omega(n) \stackrel.\approx \log\log n \tag 1$$
This means that $\log\log n$ is a normal order of $\Omega(n)$, where $\omega(n)$ denotes the number of distinct prime factors of $n$, and $\Omega(n)$ denotes the number of prime factors of $n$ with multiplicity. For $(1)$, see for example this question.
To get the overall time complexity of 4.79, let's assume that the complexity of multiplying 2 elements of $\Bbb Z/ n\Bbb Z$ is
$$\mathcal T(u\cdot v \text{ mod } n) = \mathcal O(\log^2n)$$
and exponentiation with an exponent $0\leqslant k \leqslant n$ costs
$$\mathcal T(u^k \text{ mod } n) = \mathcal O(\log^3n)$$
As explained above, a good estimator for the total number of loops in 4.79 (2. together with 2.3) is about $\Omega(n)$, and each loop body performs arithmetic no more expensive than modular exponentiation. Hence the overall complexity is about
$$\mathcal T(4.79) = \mathcal O(\log^3n\cdot\log\log n)$$