Given $a,b,c\geq0$ such that $a+b+c=3$, find the minimum value of $$P=a^2+b^2+c^2+2abc.$$
It seems like the minimum value of $P$ is $5$ when $a=b=c=1$, but I can find at least one example where $P<5$.
My attempt:
Without loss of generality, I can suppose that $a\geq b\geq c$ and so $a\geq 1$. Therefore I have: $$P\geq a^2+b^2+c^2+2bc=a^2+(b+c)^2\geq \frac{(a+b+c)^2}{2}=\frac{9}{2}.$$
The issue with this is that equality doesn't occur with this method.
What's your take on the problem?
Your attempt already contains very good observations, and is quite close to a complete solution. All that remains is an analysis of when both inequalities are equalities. You use the following two inequalities: \begin{eqnarray*} a^2+b^2+c^2+2abc&\geq&a^2+b^2+c^2+2bc\tag{1}\\ a^2+(b+c)^2&\geq&\frac{(a+b+c)^2}{2}\tag{2} \end{eqnarray*} The first inequality is an equality if and only if $abc=bc$, i.e. if and only if either $a=1$ or $bc=0$.
The second inequality is an equality if and only if $a=b+c$. Then from $a+b+c=3$ it follows that $a=\tfrac32$. Then for the first inequality to be an equality we must have $bc=0$, and hence from $b\geq c$ it follows that then $c=0$. Because $a+b+c=0$ it finally follows that $b=\tfrac32$, and so $$(a,b,c)=(\tfrac32,\tfrac32,0).$$
This shows that the minimum of $P$ is indeed $\tfrac92$, and that it is attained precisely at the points $$(a,b,c)=(0,\tfrac32,\tfrac32),\qquad(a,b,c)=(\tfrac32,0,\tfrac32),\qquad(a,b,c)=(\tfrac32,\tfrac32,0).$$
Original answer, by brute force and elementary methods:
Plugging in $c=3-a-b$, we want the minimum of $$P=2a^2+2b^2+9-6a-6b+8ab-2a^2b-2ab^2,$$ with the restriction that $a,b\geq0$ and $a+b\leq3$. At the boundary points we either have $a=0$ or $b=0$ or $a+b=3$, and hence correspondingly either \begin{eqnarray*} P&=&2b^2-6b+9,\\ P&=&2a^2-6a+9,\\ P&=&2a^2+2(3-a)^2+9-6a-6(3-a)+8a(3-a)+2a^2(3-a)+2a(3-a)^2\\ &=&2a^2-6a+9, \end{eqnarray*} where in each case $0\leq a,b\leq3$. From here the local minima on the boundary are easily determined to be at $$(a,b)=(0,\tfrac32),\qquad(a,b)=(\tfrac32,0),\qquad(a,b)=(\tfrac32,\tfrac32)$$ each with value $\tfrac92$. For the extrema of $P$ on the interior we compute the derivatives of $P$ w.r.t. $a$ and $b$, which shows that $$4a-6+8b-4ab-2b^2=0\qquad\text{ and }\qquad 4b-6+8a-2a^2-4ab=0,$$ at any interior extremum $(a,b)$ of $P$. Taking the difference shows that $$0=2(a^2-b^2)-4(a-b)=2(a-b)(a+b-2)$$ so either $a=b$ or $a+b=2$. If $a=b$ then the quadratics above both become $$0=4a-6+8b-4a^2-2a^2=6(a-1)^2,$$ which shows that $a=b=1$, and then $P=5$, which is not minimal. If $a+b=2$ then $$0=4(2-a)-6+8a-2a^2-4a(2-a)=2(a-1)^2,$$ which again yields $a=b=1$, again not yielding a minimum.
In conclusion the minimum of $P$ is $\tfrac92$, and it is attained precisely at the points $$(a,b,c)=(0,\tfrac32,\tfrac32),\qquad(a,b,c)=(\tfrac32,0,\tfrac32),\qquad(a,b,c)=(\tfrac32,\tfrac32,0).$$