How to know you need to use sin or cos?

Question

From the vectors dot product formula we know that we need cos for the correct dot product result. But how someone decide that for this case we need exactly cos and not sin?

Answer 1

All of these simply come from the geometry you are dealing with, and the different definitions of sine and cosine. In particular that sine is $\frac{\text{opposite}}{\text{hypotenuse}}$ and cosine is $\frac{\text{adjacent}}{\text{hypotenuse}}$.

How exactly to show that the inner product uses the cosine depends on how you define the inner product. There are many different ways. Indeed, in physics, it is common to define the inner product exactly by the formula $\vec u\cdot \vec v = \|\vec u\|\|\vec v\|\cos \theta$. A somewhat less contrived definition is that the inner product of $\vec u$ and $\vec v$ is the length of $\vec u$ times the length of the projection $\vec v'$ of $\vec v$ onto the line of $u$:

$$\vec u \cdot \vec v := \|\vec u\|\|\vec v'\|$$

From the triangle, and recalling that cosine is $\frac{\text{adjacent}}{\text{hypotenuse}}$, $$\cos \theta = \frac{\|\vec v'\|}{\|\vec v\|}$$ so $\|\vec v'\| = \|\vec v\|\cos\theta$, and therefore we get the formula $$\vec u\cdot\vec v = \|\vec u\|\|\vec v\|\cos \theta$$

It is cosine instead of sine because $\vec v'$ is the adjacent side, not the opposite side, of that triangle.

Similar remarks apply to the formulas for converting pitch, roll and yaw to $x,y,z$ coordinates. In each case, you build a right triangle with the indicated angle, and whether you need to use sine or cosine or tangent depends on which side of that triangle you already know, and which one you are trying to figure out. Exactly how that calculation goes depends on which definition of "pitch", "roll", and "yaw" you are using.

Answer 2

It comes from the so-called law of cosines:

Imagine the following triangle with sides of length $||\mathbf{a}||$, $||\mathbf{b}||$ and $||\mathbf{a-b}||$ (double vertical bars denote the magnitude of a vector):

According to the law of cosines, we have the following relationship: $$\require{cancel} ||\mathbf{a-b}||^2=||\mathbf{a}||^2+||\mathbf{b}||^2-2||\mathbf{a}||||\mathbf{b}||\cos{\theta}\implies\\ \left(\sqrt{(a_1-b_1)^2+(a_2-b_2)^2}\right)^2= \left(\sqrt{a_1^2+a_2^2}\right)^2+\left(\sqrt{b_1^2+b_2^2}\right)^2-2||\mathbf{a}||||\mathbf{b}||\cos{\theta}\implies\\ (a_1-b_1)^2+(a_2-b_2)^2-a_1^2-a_2^2-b_1^2-b_2^2=-2||\mathbf{a}||||\mathbf{b}||\cos{\theta}\implies\\ \cancel{a_1^2}-2a_1b_1+\cancel{b_1^2}+\cancel{a_2^2}-2a_2b_2+\cancel{b_2^2}-\cancel{a_1^2}-\cancel{a_2^2}-\cancel{b_1^2}-\cancel{b_2^2}=-2||\mathbf{a}||||\mathbf{b}||\cos{\theta}\implies\\ -2(a_1b_1+a_2b_2)=-2||\mathbf{a}||||\mathbf{b}||\cos{\theta}\implies\\ a_1b_1+a_2b_2=||\mathbf{a}||||\mathbf{b}||\cos{\theta}\implies\\ \mathbf{a}\cdot\mathbf{b}=||\mathbf{a}||||\mathbf{b}||\cos{\theta}. $$

And that is your formula for the dot product of two vectors.