I struggle with the presence of an unknown in a Markov Chain and the irreducibility concept.
I need to find the value of $a$ for which a Markov Chain with three states and transition probability matrix
- Irreducible?
- Is it aperiodic?
I know that if a Markov Chain is irreducible that we can reach regardless to the present state any other state in finite time. For me it looks like It is the case, but it there a mathematical proof of it ?
For the periodicity, I understand it has to do with the fact to go back to our initial state but it involve greatest common divisor in it and I don't quite understand the way it works.
This is not a complete answer, just a hint.
Irreducibility means that if you draw the directed graph with an edge from $i$ to $j$ iff $p_{ij}>0$, the resulting graph is strongly connected. Thus the only important thing is whether elements of the transition matrix are zero. So with your unknown there, you want to look at the cases $a=0,a=0.7$ and all other values of $a$.
When $a=0$, you can go from 1 to 2, 1 to 3, 2 to 1, 2 to 2, 3 to 2, and 3 to 3. Draw this graph. Is it strongly connected?
When $a=0.7$ you lose the $1$ to $3$ edge and gain a $1$ to $1$ edge. Is the graph strongly connected?
When $a$ is neither of those you have both the $1$ to $3$ edge and the $1$ to $1$ edge. Is the graph strongly connected?
Periodicity cares about the possible lengths of paths from a state to itself. A state $i$ is periodic if all paths from $i$ to $i$ have a length which is a multiple of some integer $T>1$. It is useful to note that "typically" a chain is aperiodic, so that your first instinct should be to try to write down two paths from $i$ to $i$ with coprime lengths for each $i$. In the presence of self-loops, this is usually easy: write down a path which never uses a self-loop (like 3->2->3), and the "same" path but using a self-loop once (like 3->2->2->3). These will have some length $n$ and $n+1$, which are coprime, so that the state in question will be aperiodic.
In fact you don't need to do this for each $i$. This is because of a useful lemma that you should either have in your notes or prove for yourself: periodicity of a state is a class property. In other words, all states in a given communicating class are either periodic or aperiodic. In particular, an irreducible chain is periodic (resp. aperiodic) if one state is periodic (resp. aperiodic).