Let $H\leq G$ be finite groups and $\chi$ an irreducible character of rep $\rho$ of $G.$ Decompose $Res_H^G\chi=a_1\psi_1+\dots+a_r\psi_r$ with $\psi_i\in Irr(H).$ I want to show $a_1^2+\dots+a_r^2\leq\frac{|G|}{|H|}$ and determine when equality occurs.
To see the inequality, I use $\sum a_i^2=\langle Res_H^G\chi,Res_H^G\chi\rangle_H=\langle\chi,Ind_H^GRes_H^G\chi\rangle_G$ by Frobenius reciprocity, and note $\text{dim}Ind_H^GRes_H^G\rho=\frac{|G|}{|H|}\text{dim}\rho$ so $\rho$ is contained at most $|G|/|H|$ times in $Ind_H^GRes_H^G\rho$ as desired.
When does equality occur? I think perhaps some Mackey theory is needed, but all I've been told about that is how $Res_H^GInd_H^G\rho'$ behaves for $\rho'$ rep of $H$, while I want to restrict first and then induce.
Let $\psi \in Irr(H)$ and assume that $\psi^G=\sum_{i=1}^sd_i\chi_i$, where $\chi_i \in Irr(G)$ and $d_i \in \mathbb{Z}_{\geq 0}$. Since the irreducible characters form an orthonormal basis, it follows that the inner product $[\psi^G,\psi^G]=\sum_{i=1}^sd_i^2:=m$. By Frobenius Reciprocity, $[\psi^G,\psi^G]=[(\psi^G)_H,\psi]$. So $(\psi^G)_H=m\psi + \Delta$, where $\Delta$ is a character of $H$ or $\Delta \equiv 0$. Hence $(\psi^G)_H(1)=\psi(1)|G:H| \geq m\psi(1)$, and so $m \leq |G:H|$.
Conversely, assume $\chi_H=\sum_{j=1}^te_j\psi_j$, where $\psi_j \in Irr(H)$ and $e_j \in \mathbb{Z}_{\geq 0}$. It follows that $[\chi_H,\chi_H]=\sum_{j=1}^te_j^2:=n$. But $[\chi_H,\chi_H]=[(\chi_H)^G,\chi]$, so $(\chi_H)^G=n\chi + \Gamma$, where $\Gamma$ is a character of $G$ or $\Gamma \equiv 0$. Hence $(\chi_H)^G(1)=\chi(1)|G:H| \geq n\chi(1)$, and so $n \leq |G:H|$.
I leave it to you to show that equality occurs in the second case if and only if $\chi$ vanish outside $H$. In the first case this happens for instance when $H$ is normal, $\psi$ is invariant and $s=1$.