First of all, I know how to solve the following exercise; the problem is that there is no solution. "In RSA, Alice chooses $p=53$, $q=63$, public key ($n=3339, e=13$). When Bob sends the message $m=5$, what is the message that Alice will read?"
$φ(n) = 3224$
For encoding I have $ C = 5^{13} \mod(3339) = 1454$
The problem start's when I decode the message: First I have to calculate the secret key $d$ s.t. $$13d = 1 \mod(3224)$$, for which there is no solution (I know that $q$ should be prime and I assume this is the problem). I contacted my professor and the answer i got is that the mistake is on purpose and have to comment what is going on. I really do not understand what i should answer.
Just so this question has an answer...
There are two problems here. The first is that the Euler totient $\phi(n)$, for $n = pq$, equals $(p-1)(q-1)$ if and only if $p$ and $q$ are two distinct prime. Here we have $q = 63 = 9\times 7$ so $q$ is not prime.
We have in fact $n = 53\times 63 = 53\times 7\times 3^2$ so $\phi(n) = \phi(53)\phi(7)\phi(3^2) = 52\times 6\times 6 = 1872$.
The second is that the public exponent $e = 13$ is not relatively prime with $\phi(n) = 1872 = 2^4\times 3^2\times 13$, so there is no $d$ such that $de\equiv 1 \pmod{\phi(n)}$.