Let's say the input of a system is expressed as $$x(t)=\sum_{k=-\infty}^{k=\infty}a_ke^{jkω_0t}$$ and that we can find the coefficients $a_k$. Our input goes through a filter $h(t)$. I found that you can calculate the coefficients $b_k$ of the output using the following $$b_k=a_kH(kω_0) $$ I am confused a little by this. I've solved some other problems in which I make use of the linearity property, that is finding $b_k$ from $a_k$ multiplying by the coefficient of the new function. However, in this example the multiplication in the time domain would be a convolution and not just a multiplication.
I can see that he's using $H(ω)$ , is that a property I'm not aware of? Can he use the coefficients $a_k$ in the frequency domain to find $b_k$?
Consider the simplest case where the input signal is a single exponential, i.e., $$ x(t)=a_k e^{ik\omega_0 t}. $$
The (time-domain) output of the filter is, of course, the convolution of $x(t)$ and $h(t)$, which can be computed as
$$ \begin{align} y(t) &= x(t) * h(t)\\ &=\int_{-\infty}^{\infty}h(\tau)x(t-\tau)d\tau \\ &=\int_{-\infty}^{\infty}h(\tau)a_k e^{ik\omega_0(t-\tau)}d\tau \\ &=a_k e^{ik\omega_0 t}\int_{-\infty}^{\infty}h(\tau) e^{-ik\omega_0 \tau}d\tau \\ &=a_k e^{ik\omega_0 t}H(k\omega_0), \\ \end{align} $$
where $H(\omega) \triangleq \int_{-\infty}^{\infty}h(\tau) e^{-i\omega \tau}d\tau$ is the filter frequency response. You may now generalize to the case where $x(t)$ is a sum of exponentials by using the linearity of the convolution operator.
Another way to see this is to note that the Fourier transform of $x(t)$ is an impulse train $$ X(\omega) = \sum_{k=-\infty}^{\infty}a_k \delta(\omega - k\omega_0) $$
Therefore, the Fourier transform of the output signal should be
$$ \begin{align} Y(\omega) &= X(\omega) H(\omega)\\ &= \sum_{k=-\infty}^{\infty}a_k \delta(\omega - k\omega_0) H(\omega)\\ &= \sum_{k=-\infty}^{\infty}a_k \delta(\omega - k\omega_0) H(k \omega_0),\\ \end{align} $$
where the last equality follows by fundamental properties of the delta function. Clearly, the last expression implies that $y(t)$ is also a sum of the same exponentials as $x(t)$ weighted by $b_k=a_k H(k\omega_0),k\in\mathbb{Z}$.