My professor provided me the formula:
$$
\cos(\theta)= \dfrac{\langle v,u\rangle}{ ||v|| \cdot ||u||}
$$
and asked us to compute the cosine of the angle between $h(x)= \cos(x/4)$ and $k(x)=\sin(x/4)$ on the interval $[0, \pi]$.
He said one can define the inner product of $h$ and $k$ by the integral from zero to $\pi$ of $h(x)k(x) dx$.
Does anybody know how I can approach this problem?
You don't want the integral from one to $\pi$ but from zero to $\pi$, since that is the domain of your functions.
The idea is that you are defining an inner product by the formula $$\langle f, g\rangle:=\int_0^\pi f\cdot g$$ This inner product, defined on a(n infinite-dimensional) vector space of functions, is what allows you to think geometrically. In particular, you can now use this inner product to define lengths (aka norms, aka magnitudes) by $$\|f\|:=\sqrt{\langle f, f\rangle}$$ and angles by $$\cos\theta:=\frac{\langle f, g\rangle}{\|f\|\cdot\|g\|}$$ This point of view generalizes the familiar dot-product geometry you learned in regular $n$-dimensional space. You should no longer think of your vectors as an ordered list of numbers, though, because the vectors in these spaces have infinitely many "components." But that's okay, because the inner product we defined above doesn't require us to "dot" respective components. Instead of summing up the products of the respective components, as we do with the dot product in regular Euclidean space, we now integrate the products over an entire interval.