Let $H \le G$ and let $\chi$ be a character of $G$ which vanishes on $G-H$. Assume that either $H=1$ or $G$ is abelian. Show $|G:H| | \chi(1)$.
Hint: Let $\lambda $ be an irreducible constituent of $\chi_H$. Under either hypothesis, find $\mu \in Irr(G)$ with $\mu_H = \lambda$. Compute $[\chi, \mu] $ and conclude $|G:H| |[\chi_H , \lambda ] $.
I am a bit lost at how the hint works:
(0) If $H=1$ then I think I can do it: then an irreducible constituent of $\chi_H$ is $1_H$. This is the restriction of $1_G$. We have $[\chi, 1_G] |G| = |H| [ \chi_H, 1_H]$. Thus, $|G:H| | [\chi_H, 1_H] = \chi(1)$.
(i) I am lost at how one find $\mu$ when $G$ is abelian.
(ii) I have $|G| [\chi, \mu] = |H|[\chi_H, \lambda]$, hence $|G:H| | [ \chi_H, \lambda] $. But I cannot end the proof from here.
Here is the fully worked answer to the Exercise (2.16) in Isaacs' book.
Proof Assume first that $H=1$. Now $|G|[\phi,1_G]=\sum_{g \in G}\phi(g)=\phi(1)$. Since $\phi$ is a character, $\phi(1) \neq 0$ and $[\phi,1_G]$ is a positive integer, it follows that $|G|$ divides $\phi(1)$. In fact, in this case $\phi=b\rho$, where $b=\frac{\phi(1)}{|G|}$ and $\rho$ is the regular character of $G$, see Exercise (3.8) of the same book.
Now assume that $G$ is abelian and $\phi\equiv0$ outside $H$. We can write $\phi=\sum_{\mu \in Irr(G)} a_{\mu}\mu$, with $a_{\mu}$ non-negative integers and not all equal to $0$. Note that the $\mu$ are of course all linear since $G$ is abelian. Hence their restrictions $\mu_H$ are also linear and irreducible. This implies that $\phi_H=\sum_{\mu \in Irr(G)} a_{\mu}\mu_H$ is the decomposition of $\phi_H$ as a sum of irreducible characters of $H$. Note that different $\mu's$ can be equal when restricted to $H$. So in this last sum some of the irreducible constituents may coincide.
Put $\phi_H=\sum_{\lambda \in Irr(H)} b_{\lambda}\lambda$ with $b_{\lambda}$ non-negative integers. Fix such $\lambda$, then $\lambda=\mu_H$ for some irreducible constituent $\mu$ of $\phi$. Since $\phi$ vanishes off $H$, $$b_{\lambda}|H|=\sum_{h \in H}\phi(h)\overline{\lambda(h)}=\sum_{g \in G}\phi(g)\overline{\mu(g)}=|G|a_{\mu}$$
So we see that each of the $b_{\lambda}=|G:H|a_{\mu}$ is hence divisible by $|G:H|$. Since $\phi(1)=\sum_{\lambda \in Irr(H)} b_{\lambda}\lambda(1)=\sum_{\lambda \in Irr(H)} b_{\lambda}$, we now see that $|G:H|$ divides $\phi(1)$, as wanted.$\square$
Note 1 A natural common generalization of the situation $H=1$ and $G$ is abelian is $H \subseteq Z(G)$. However, under these hypotheses the above result is not true. By the way, in this situation, owing to Lemmata (2.27)(c) and (2.29) of Isaacs' book, $\phi(1)^2=|G:H|[\phi,\phi]$, hence $|G:H|$ divides $\phi(1)^2$.
Here is a counterexample that shows that $|G:H|$ not necessarily divides $\phi(1)$. The character table of the quaternion group $Q$ of order $8$ looks as follows. Here $Q=\{\pm 1,\pm i, \pm j,\pm k \}$, where $i^2=j^2=k^2=-1, ij=k,jk=i,ki=j$. $Q$ has five conjugacy classes.
$$ \begin{array}{c|c|c|c|c|c} \hline &1& -1 & \{\pm i \} & \{\pm j \} & \{\pm k \} \\ \hline 1_Q & 1 & 1 & 1 & 1 & 1 \\ \hline \chi_2 & 1 & 1 & 1 & -1 & -1 \\ \hline \chi_3 & 1 & 1 & -1 & -1 & 1 \\ \hline \chi_4 & 1 & 1 & -1 & 1 & -1 \\ \hline \chi_5 & 2 & -2 & 0 & 0 & 0 \\ \hline \end{array} $$
So, $\chi_5$ vanishes off the center $Z(Q)=\{\pm 1\}$. But $|Q:Z(Q)|=4$ does not divide $\chi_5(1)=2$.
Note 2 The above lemma is being used in the proof of Theorem (3.13).