Determine all the pairs of positive odd integers $(a,b),$ such that $a,b>1$ and $7\varphi^2(a)-\varphi(ab)+11\varphi^2(b)=2(a^2+b^2),$ where $\varphi(n)$ is Euler's totient function. NOTE -$\varphi^2(n)$ $\neq$ $\varphi(\varphi(n))$.
Okay, since RHS is congruent to 4 mod 8 so LHS is congruent to 4 mod 8. Now, we can bound number of factors of $ab$ by this fact so I got $a=p^{k+1}\cdot q^{l+1}$ and $b=p^{r+1}\cdot q^{s+1}$ where p,q are primes such that 2<p<q and k,l,r,s are non negative integers. It can be proven that $l=s$ by some $v_q$ argument. Now, talking $v_p$ we will get $k=r$ iff we have $p\not| q-1$ but if $k=r$ then we will $pq$ is a multiple of 18 but no $\varphi(pq)$ is congruent to 4 mod 8 or even 0 mod 8 so $p\vert q-1$. Okay, everything was going well but suddenly I saw a example (a,b)=(15,3) which contradicts my assumption that $p\vert q-1$. Now, how can I prove that there is atleast one number of the two of the form $q^x$ and move further. Please help!
As you note, the right hand side is congruent to $4$ mod $8$ because $a$ and $b$ are odd. Then the left hand side is also congruent to $4$ mod $8$. Because $a$ and $b$ are odd, for each prime $p$ dividing either $a$ or $b$ the factor $p-1$ dividing either $\varphi(a)$ or $\varphi(b)$ is even. Now you can distinguish two cases:
If $a$ and $b$ are both divisible by at least two distinct prime numbers then $$\varphi(a)^2\equiv\varphi(b)^2\equiv0\pmod{8}.$$ It follows that $\varphi(ab)\equiv4\pmod{8}$, so $ab$ is divisible by at most two distinct prime numbers, and hence $$a=p^{a_p}q^{a_q}\qquad\text{ and }\qquad b=p^{b_p}q^{b_q},$$ for some pair of prime numbers $p$ and $q$. Then you can plug this into the equation $$7\varphi(a)^2-\varphi(ab)+11\varphi(b^2)=2(a^2+b^2),$$ to find that $p$ divides $77(q-1)$ and $q$ divides $77(p-1)$. In particular, without loss of generality $p\in\{7,11\}$. From here it is a matter of doing some slightly cumbersome algebra.
On the other hand, if $a$ and $b$ are not both divisible by at least two distinct prime factors, then at least one of $a$ and $b$ is a prime power. So without loss of generality $a=p^{a_p}$ for some prime number $p$. Again you can plug this into the equation $$7\varphi(a)^2-\varphi(ab)+11\varphi(b^2)=2(a^2+b^2),$$ where you may want to distinguish further cases depending on whether $p$ divides $b$ or not.