I would appreciate if somebody could help me with the following problem
Q:$P(x,y)$ given point lie inside or on the boundary of triangle $\triangle ABC$.
Find maximum and minimum $f(a,b,c)$
$$f(P)=\vec{PA}\cdot \vec{PB}+\vec{PB}\cdot \vec{PC}+\vec{PC}\cdot \vec{PA}$$

Let us rewrite $f$ in terms of position vectors. For this purpose consider the following
$$\eqalign{ & f = \overrightarrow {PA} .\overrightarrow {PB} + \overrightarrow {PB} .\overrightarrow {PC} + \overrightarrow {PC} .\overrightarrow {PA} \cr & \overrightarrow {PA} .\overrightarrow {PB} = \left( {\overrightarrow P - \overrightarrow A } \right).\left( {\overrightarrow P - \overrightarrow B } \right) = {\left| {\overrightarrow P } \right|^2} - \left( {\overrightarrow A + \overrightarrow B } \right).\overrightarrow P + \overrightarrow A .\overrightarrow B \cr & \overrightarrow {PB} .\overrightarrow {PC} = \left( {\overrightarrow P - \overrightarrow B } \right).\left( {\overrightarrow P - \overrightarrow C } \right) = {\left| {\overrightarrow P } \right|^2} - \left( {\overrightarrow B + \overrightarrow C } \right).\overrightarrow P + \overrightarrow B .\overrightarrow C \cr & \overrightarrow {PC} .\overrightarrow {PA} = \left( {\overrightarrow P - \overrightarrow C } \right).\left( {\overrightarrow P - \overrightarrow A } \right) = {\left| {\overrightarrow P } \right|^2} - \left( {\overrightarrow C + \overrightarrow A } \right).\overrightarrow P + \overrightarrow C .\overrightarrow A \cr} $$
Hence, you may have
$$f = 3\overrightarrow P .\overrightarrow P - 2\overrightarrow P .\left( {\overrightarrow A + \overrightarrow B + \overrightarrow C } \right) + \left( {\overrightarrow A .\overrightarrow B + \overrightarrow B .\overrightarrow C + \overrightarrow C .\overrightarrow A } \right)$$
Now compute the gradient vector and hessian matrix of $f$ to find
$$\eqalign{ & \nabla f = 6\overrightarrow P - 2\left( {\overrightarrow A + \overrightarrow B + \overrightarrow C } \right) \cr & {\bf{H}} = \nabla \nabla f = 6{\bf{I}} \cr} $$
As you see, at the critical point the gradient is zero
$$\nabla f = 0\,\,\,\,\,\,\,\, \to \,\,\,\,\,\,\,\,\overrightarrow P = {1 \over 3}\left( {\overrightarrow A + \overrightarrow B + \overrightarrow C } \right)$$
but since the hessian matrix is positive definite you have a local minimum. But this is a quadratic form and hence the local minimum is the global minimum. Finally, you may conclude that the minimum of $f$ happens when $\overrightarrow P $ is at the centroid of the triangle.