Let $ABCD$ be a square. Suppose $P$ be a point strictly inside the square such that $AP = 5$ and $BP = 13$. How many distinct integer values are possible for the area of $ABCD$?
The possible choices were: $143, 144, 179, 180, 181$
If you start with the triangle equality then the minimum side would be $\sqrt{65}$ and the maximum side length would be $\sqrt{323}$. However, that is too many possibilities. I also thought that the triangle had to be acute, which means that the maximum side length had to be less than $\sqrt{169+25}$. But this is not true either. I am not sure how to proceed further.
You're on the right approach about determining the range for AB. As you mentioned, we can determine the maximum value by triangle inequality. For now, don't worry about the integer requirement for area, and the $<$ ensures P is strictly within the square. $$AB < AP + BP = 18$$
Now for minimum value. Triangle inequality unfortunately cannot be used here, as at the predicted minimum ($AB = BP - AP = 8$), $\angle PAB$ is obtuse. While obtuse triangles are possible within the square (e.g. when AB is just less than 18), the angles involving the square's side ($\angle PAB$ and $\angle PBA$) cannot be obtuse, otherwise $P$ will lie outside the square (as Jyrki mentioned).
So the actual minimum occurs when the triangle is almost right-angled at either $\angle PAB$ or $\angle PBA$. Since BP > AP, BP must be the hypotenuse and $\angle PAB$ the right-angle. Then the minimum of AB can be calculated via Pythagoras theorem (as Ross implied), $$AB > \sqrt{BP^2 - AP^2} = 12$$
So now we have the range for AB, $$12 < AB < 18$$
But the question wants area, whose range is $$12^2 < AB^2 < 18^2$$
Then the number of integer values for area is, $$18^2 - 12^2 - 1 = 179$$
1 is subtracted to ensure both ends are excluded, since P lies on the square at those values.