I know that $P_K$ can be obtained as follows:
Assume that $\rho$ is $\lambda/\mu$ and that the probability that there are $m$ customer in the system is $P_m$.
$P_0+P_1+...+P_K=1$
⇔ $P_0+\rho P_0+...+\rho^K P_0=1$
⇔ $P_0(1+\rho+...+\rho^K)=1$
⇔ $P_0=\frac{1-\rho}{1-\rho^{K+1}}$
Then, $P_m=\rho^m P_0 = \frac{\rho^m(1-\rho)}{1-\rho^{K+1}}$ and
$P_K=\frac{\rho^K(1-\rho)}{1-\rho^{K+1}}$.
Q1. Is the formulation above true even when the value of $\rho > 1$?
Q2. (If the answer to Q1 is yes) Why is the $P_K$ value not 1 when $\rho > 1$? For example, if $K=3$ and $\rho=1.5$, then $P_K=0.4154$. I think that the value should be 1 because of $\lambda > \mu$.
Really really thanks a lot for your kindness.
Yes, that expression is correct even when $\rho \stackrel{\text{def}}{=} \lambda/\mu \geq 1$. Keep in mind that $\rho$ here is not the utilization—at least not in the sense that it represents the average number of busy servers.
When the queue is saturated, the server is busy, and the customer in service departs at rate $\mu$. Once that customer departs, and there is room for one additional customer, the vacant spot in the queue is only filled at rate $\lambda$—not at an infinite rate. That might give some intuition as to why $P_K \not= 1$; in fact, it's not even generally the case that the server is busy with probability $1$, as represented by $P_0 > 0$ if $\lambda > 0$.
A simple sanity check is represented by the case $K = 1$, where there is no room for any queue (other than the customer in service). We then have a two-state system with birth rate $\lambda$ and death rate $\mu$, and clearly
$$ P_0 = \frac{\mu}{\lambda+\mu} = \frac{1}{1+\rho} $$ $$ P_1 = \frac{\lambda}{\lambda+\mu} = \frac{\rho}{1+\rho} $$
This is true even when $\rho \geq 1$, and it fits your general expression.