The original question is :
Given three points on the xy plane, O(0,0) , A(1,0) , B(-1,0). Point P is moving on the plane satisfying the condition $(\vec{PA}\cdot\vec{PB})+3(\vec{OA}\cdot\vec{OB})=0$ . If the maximum and minimum values of $|{\vec{PA}}|| {\vec{PB}}|$ are M and m respectively then find the value(s) of $M^2+m^2$.
I found the solution to the exact problem $\color{red}{here.}$
But I approached the question in a different way.
My approach:
$$\vec{OA}=(1,0)$$ $$\vec{OB}=(-1,0)$$ Thus, $$\vec{OA}\cdot\vec{OB}=-1$$ Hence, according to the condition in the question: $$\vec{PA}\cdot\vec{PB}=3$$
Therefore, $$|\vec{PA}||\vec{PB}|\cos(\theta)=3$$ Or $$|\vec{PA}||\vec{PB}|=\frac{3}{\cos(\theta)}$$ Where $\theta$ is the angle between vectors $\vec{PA}$ and $\vec{PB}$.
So according to what I've got, maximum value of $|\vec{PA}||\vec{PB}|$ occurs when $\theta=90°$ .
This contradicts the claim in the above given link that the maximum value of the required expression is 5.
So I want to know why is this contradiction arising,as I don't see any mistake in the answer given in the above link as well.
Thanks in advance.
If you could construct two arbitrary vectors $v$ and $w$ of any lengths in any directions, then indeed there would be no maximum value of $\lVert v\rVert \lVert w\rVert$, because you could make $\theta$ as close to a right angle as you want and then make the vectors long enough so that $v\cdot w=3$.
But those are not the conditions under which you have to solve this problem.
If $\theta$ is near $90^\circ$ then $P$ is nearly on a circle whose diameter is the line segment $AB$. You cannot make the product $\lVert \vec{PA}\rVert \lVert \vec{PB}\rVert$ much larger than $2$. And of course $\vec{PA}\cdot\vec{PB}$ will be less than $3.$
Although your formula is true, you must honor the other constraints of the problem statement. That’s why you cannot maximize $\lVert \vec{PA}\rVert \lVert \vec{PB}\rVert$ by setting $\theta$ to $90^\circ$ or close to $90^\circ$.