For a triangle with standard conventions, prove the inequality in the title, that is: $$\frac{h_a}{l_a} + \frac{h_b}{l_b} + \frac{h_c}{l_c} \leq \frac{\sqrt{3(p^2 + r^2 - 8Rr)}}{2R} + \sqrt{3\left(1 - \frac{r}{2R}\right)}$$
I observed the fact that: $$\frac{h_a}{l_a} = \sin \left(\pi - \frac{A}{2} - B\right) = \sin \left(\frac{A}{2} + B\right) = $$ $$= \sin \frac{A}{2} \cos B + \cos \frac{A}{2} \sin B = $$ $$2 \sin \frac{A}{2} \left(2\cos^2 \frac{B}{2} - 1\right) + 2\sin \frac{B}{2} \cos \frac{B}{2} \cos \frac{A}{2}$$
So, we have that: $$\text{LHS} = 4 \sum_{cyc} \sin \frac{A}{2} \cos^2 \frac{B}{2} + 2 \sum_{cyc} \sin \frac{A}{2} \cos \frac{A}{2} \cos \frac{C}{2} - 2 \sum_{cyc} \sin \frac{A}{2}$$
But now, I am not able to use any formulae to transform $p, r, R$ expressions intotrigonometrical ones, so that I could apply trigonometric inequalities.
Notation
To make formulas more concise, I'll use some notation.
Left Side
$$ \frac{h_a}{l_a} = \cos\left(\frac{\beta - \gamma}{2}\right),\ \frac{h_b}{l_b} = \cos\left(\frac{\alpha - \gamma}{2}\right),\ \frac{h_c}{l_c} = \cos\left(\frac{\alpha - \beta}{2}\right). $$
$$ \frac{h_a}{l_a} + \frac{h_b}{l_b} + \frac{h_c}{l_c} = \dot b \dot c + bc + \dot a \dot c + ac + \dot a \dot b + ab. $$
Right Side
By the law of sines: $\bar a = 2RA, \bar b = 2RB, \bar c = 2RC$.
We will need some trigonometric identities (these depend only on fact that $\alpha + \beta + \gamma = \pi$):
Identity1. $A + B + C = 4 \dot a \dot b \dot c$
(proof omitted)
Identity2. $\dot A + \dot B + \dot C = 4 abc + 1$
(proof omitted)
Semiperimeter
$$p= \frac12 (\bar a + \bar b + \bar c) = R(A + B + C) = 4 R \dot a \dot b \dot c.$$
Expression $\frac rR$
Area of the triangle $S = \frac12 \bar a \bar b C = 2R^2 ABC$.
Another formula for area of the triangle is $S = r p$, and also remember that $A = 2a \dot a,\ B = 2b \dot b,\ C = 2c \dot c$, therefore we get:
$$ \frac rR = \frac S{pR} = \frac{2R^2 ABC}{4 R^2 \dot a \dot b \dot c} = \frac{2 a \dot a \cdot 2 b \dot b \cdot 2 c \dot c}{2 \dot a \dot b \dot c} = 4abc. $$
In one place we will use this expression, in other two places we will use a different expression. Remember Identity2 and double-angle formulae for cosine $\dot A = 1 - 2a^2,\ \dot B = 1 - 2b^2,\ \dot C = 1 - 2c^2$:
$$ \frac r{2R} = 2abc = \frac12 (\dot A + \dot B + \dot C - 1) = 1 - (a^2 + b^2 + c^2). $$
Expression $\frac{\sqrt{3(p^2 + r^2 - 8Rr)}}{2R}$
Now we will be able to get a nice expression for first square root in the inequality:
$$ \frac{\sqrt{3(p^2 + r^2 - 8Rr)}}{2R} = \frac {\sqrt3}2 \sqrt{\left( \frac pR \right)^2 + \left( \frac rR \right)^2 - 8 \frac rR } = \frac {\sqrt3}2 \sqrt{16(\dot a \dot b \dot c)^2 + 16 (abc)^2 - 16 (1 -(a^2 + b^2 + c^2) )} = 2 \sqrt3 \sqrt{(\dot a \dot b \dot c)^2 + (abc)^2 - 1 + (a^2 + b^2 + c^2) } $$
Let's use Pythagorean identity ($\dot a^2 + a^2 = 1,\ \dot b^2 + b^2 = 1,\ \dot c^2 + c^2 = 1$):
$$ (\dot a \dot b \dot c)^2 = (1 - a^2) (1 - b^2) (1 - c^2) = 1 - (a^2 + b^2 + c^2) + (ab)^2 + (ac)^2 + (bc)^2 - (abc)^2. $$
In square root many summands cancel out, and what is remaining is:
$$ \frac{\sqrt{3(p^2 + r^2 - 8Rr)}}{2R} = 2 \sqrt3 \sqrt{(ab)^2 + (ac)^2 + (bc)^2}. $$
Expression $\sqrt{3\left(1 - \frac{r}{2R}\right)}$
$$ \sqrt{3\left(1 - \frac{r}{2R}\right)} = \sqrt3 \sqrt{a^2 + b^2 + c^2}. $$
Applying AM ≤ QM
Let's use classic (arithmetic mean) ≤ (quadratic mean) inequality. We will use it two times: once for each square root.
$$ 2 \sqrt3 \sqrt{(ab)^2 + (ac)^2 + (bc)^2} = 6 \sqrt{\frac {(ab)^2 + (ac)^2 + (bc)^2}{3} } \geq 2 ab + 2 ac + 2 bc. $$
Now another square root:
$$ \sqrt3 \sqrt{a^2 + b^2 + c^2} = 3 \sqrt{\frac{a^2 + b^2 + c^2}{3}} \geq a + b + c. $$
We just proved that $a + b + c + 2ab + 2ac +2bc \leq (\text{right-hand-side})$.
In next section we'll prove, that this lower bound is actually exactly the left-hand-side, that is, we'll prove that
$$ \frac{h_a}{l_a} + \frac{h_b}{l_b} + \frac{h_c}{l_c} = a + b + c + 2ab + 2ac +2bc $$
Proving the inequality
Let's subtract two sides
$$ \frac{h_a}{l_a} + \frac{h_b}{l_b} + \frac{h_c}{l_c} - (a + b + c + 2ab + 2ac +2bc) \\ = (\dot b \dot c + bc + \dot a \dot c + ac + \dot a \dot b + ab) - (a + b + c + 2ab + 2ac +2bc) = (\dot b \dot c - bc + \dot a \dot c - ac + \dot a \dot b - ab) - (a + b + c) \\ = \cos\frac{\beta + \gamma}{2} + \cos\frac{\alpha + \gamma}{2} + \cos\frac{\alpha + \beta}{2} - (a + b + c). $$
Because $\beta + \gamma = \pi - \alpha$, it makes $\cos\frac{\beta + \gamma}{2} = \cos(\frac\pi2 - \frac\alpha2) = \sin\frac\alpha2 = a$. Similarly $\cos\frac{\alpha + \gamma}{2} = b$, $\cos\frac{\alpha + \beta}{2} = c$, which means:
$$ \cos\frac{\beta + \gamma}{2} + \cos\frac{\alpha + \gamma}{2} + \cos\frac{\alpha + \beta}{2} - (a + b + c) = a + b + c - (a + b + c) = 0, $$
which proves the inequality. QED.