We know that the sum of any two forces is always equal to $A + B + 2AB\cos\theta$
For minimum and maximum there will be a total of four cases since it is not told above whether the resultant is the addition or subtraction of the two vectors.
Case 1: Addition of two vectors
a) For maximum, $\cos\theta = 1$. Therefore $A + B + 2AB\cos\theta$. But this may become false if either $A$ or $B$ is a negative force since the force is a vector.
b) For minimum, $\cos\theta = -1$.
This is the long way, but in the solution textbook this is what they have done:
$A + B =7$ and $A - B = 3$. Therefore $A = 5N$ and $B = 2N$.
How is this the correct way to write this?
Neither do we also know if either $A$ or $B$ is +ve value or -ve value?
The sign tells us the direction.
The resultant force will be maximum when they are pointing to the same direction and minimum when they are in opposite direction.
The smaller force is $2N$.
Remark: The expression for the resultant should be $\sqrt{A^2+B^2+2AB\cos \theta}$ by cosine rule.