The total output is given by $$ \left(1-\frac{1}{N+1} \right)\frac{a-c}{b}$$ with $N=2$. The inverse demand function is $P=a-bx$ with $a=5, b=1, c=2$, where $x$ is the total output and $c$ is the cost which is a constant.
Firms 'A' and 'B' are both rational and Firm 'A' knows that firm 'B' is rational
The possible strategies(output) for 'A' are $Q_a=\{0,0.25,0.5,0.75,1,1.25,1.5,1.75,2\}$. Which of these is never chosen by 'A'? what I have done
On
$$Q^*_A = \frac{5-Q_B}{2}$$
$$Q^*_B = \frac{5-Q_A}{2}$$
Using IEDS (iterative elimination of dominant strategy) method,
$\text{Step 1}$
Note that $Q^*_A \le \frac52$ since $Q_B \ge 0$
Similarly, $Q^*_B \le \frac52$ since $Q_A \ge 0$
$\text{Step 2}$
Since, $Q^*_B \le \frac52$, and Firm $A$ knows that Firm $B$ is rational,
$\implies\frac{5-Q_B}2 \ge \frac{5-\frac52}2$
$\implies Q^*_A \ge \frac54$
Since, Firm $B$ is not aware of $A$ being rational, this is where the iterative elimination ends.
$1.25 \le Q_A \le 2.5$
In case, it is not the total cost but the marginal cost that is constant (in which case, the reaction curves will change), we get
$0.9375 \le Q_A \le 1.25$
Although, the equilibrium quantity for both the firms comes out to be $Q_A^*=Q_B^*=\frac53$ as in your attempt, this does not agree with the bound that is imposed on the total quantity.
$Q_T = Q_A + Q_B= 2$
Neither firm $A$ nor firm $B$ would choose on an output greater than half of the total quantity. So that, $Q_A \le 1$ and $Q_B \le 1$.
The profit equation for firm $A$ is:
$\pi = (5-(Q_A+Q_B))Q_A - 2 = (5-2) Q_A -2 = 3Q_A -2$
The profit cannot be less than zero, so we get a lower bound:
$\pi \ge 0 \implies Q_A \ge \frac23$
Since, $\frac23 \le Q_A \le 1$
$Q_A \in \{0.75,1\}$