An urn $U$ contains always $N$ balls, some white and some black balls. Fix $p \in ]0,1[$; at each stage a coin having probability $p$ of landing heads is flipped. If heads appear, then a ball is chosen at random from the urn and is replaced by a white ball; if tails appear, then a ball is chosen at random from the urn and is replaced by a black ball. For any $n \geq 0$ let $X_n$ denote the number of white balls in the urn after the $n^{th}$ stage.
It's easy to see that $(X_n)_{n\geq0}$ is a Markov chain on $\left\{ 0,...,N\right\}$ . How to explicit its transition matrix $Q$. Help me please. Thank you so much.
Definition of a transition matrix:
Define the state vector as the column vector ${\bf p}=(p_0,\cdots,p_N)^T$ of probabilities $p_i$ of the urn being in state $i$, i.e., of there being $i$ white balls in the urn. At each stage, the transition matrix $Q$ brings each state ${\bf p}_n$ to the new state ${\bf p}_{n+1}$ by operating as ${\bf p}_{n+1}=Q{\bf p}_n$
Say after a given stage $n$ there are $i$ white ball. The probability of taking then a white ball is $p_w=i/N$. Thus, in the following stage the probability of
Thus the transition matrix $Q$ is given by $$Q_{ij}\,=\,(1-p_w)p\,\delta_{i(j+1)}\,+\,p_w(1-p)\,\delta_{i(j-1)}\,+\,(p_wp+(1-p_w)(1-p))\,\delta_{ij}$$
That is, at each column $j$ of $Q$ the only non-zero values are the diagonal terms and one above/below them.
We can check that this is indeed a transition matrix as $$\sum_jQ_{ij}=(1-p_w)p+p_w(1-p)+p_wp+(1-p_w)(1-p)=1$$ as it should.