we consider the geometric euclidean space,
My problem seems trivial by eyesight, but I am struggling to prove it algebraically with rigor.
I am allowed to use the cosine rule and the normal definition of the angle in the euclidean vector space. Furthermore the polarisation rule for scalar products.
We have a triangle $\triangle abc$ with an inscribed triangle $\triangle \hat{a}bc$, they share the side $bc$ as their base and $b\hat{a} = ba \cdot j$, with $j \in (0,1)$.
Prove that: $\angle bc\hat{a} < bca$ or equivalently $\cos(bca) < \cos(bc\hat{a})$.
To illustrate and define properties here is my sketch:
Let's assume that $\angle BCA$ and $\angle BC\hat{A}$ are both obtuse or both acute, because otherwise $\cos(\angle BCA)$ is negative while $\cos(\angle BC\hat{A})$ is positive.
In the case that they are both acute, draw a line through $B$ perpendicular to $\overline{BC}$, and extend $\overrightarrow{BA}$ and $\overrightarrow{B\hat{A}}$ until they hit that line. Now, notice that $\cos(\angle BCA)$ and $\cos(\angle BC\hat{A})$ share the same "adjacent side" $BC$ but the hypotenuse corresponding to $CA$ is longer, meaning $\cos(\angle BCA)< \cos(\angle BC\hat{A})$.
In the case that they are both obtuse, similar logic holds but the line segments $CA, C\hat{A}$ will be extended in the other direction, and the signs are flipped. Ultimately you should get the same result (try it!).
P.S. Learning LaTeX is hard at first, but will make your future questions/answers much easier on the eyes, e.g. $j \in (0, 1)$. It would also help to place your setup and question first, then explain what you've tried so far. I've also changed all vertex labels to capital letters because it's conventional to denote vertices with capital letters and side lengths with lowercase letters in a triangle.