I'm trying to follow this paper: Flows, scaling, and the control of moment hierarchies for stochastic chemical reaction networks . Sorry for poor use of vectors, couldn't figure out how to get the vectors not to take a whole line.
Avoiding the physical details behind these matrices, there are two example matrices they give, A and Y. The examples they use are
A = \begin{bmatrix}-\alpha & \epsilon & 0 \\ \alpha & -2\epsilon & \beta \\ 0 & \epsilon & -\beta \end{bmatrix}
and Y = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}.
Now, there are two vector sets of interest, the basis of ker(YA)$^\perp$ and the basis of the quotient space ker(YA)/ker(A). Given these, I've calculated that
ker(A) = \begin{bmatrix} 1/\alpha \\ 1/\epsilon \\ 1/\beta \end{bmatrix}
and ker(YA) (there are two vectors in this basis, sorry for not having in same line) = \begin{bmatrix} 1/\alpha \\ 0 \\ 1/\beta \end{bmatrix}, \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}
So I get the basis for ker(YA)$^\perp$ = \begin{bmatrix} \alpha \\ 0 \\ -\beta \end{bmatrix},
To calculate the basis for ker(YA)/ker(A), I just found a vector orthogonal to ker(A) in the vector space of ker(YA), being = \begin{bmatrix} 1/\alpha \\ -1/\epsilon \\ 1/\beta \end{bmatrix}.
However, this answer is obviously wrong as it is not orthogonal to ker(A).
However, in the text they get different answers than the paper (page 17, the row vectors in the parentheses of the last line of equation 64),
basis for ker(YA)$^\perp$ = \begin{bmatrix} \alpha \\ 0 \\ -\beta \end{bmatrix} and
basis for ker(YA)/ker(A) = \begin{bmatrix} \alpha \beta^2 \\ -\epsilon(\alpha^2 + \beta^2) \\ \beta \alpha^2 \end{bmatrix}
And I'm not sure what I did wrong, could someone help me out with where I went wrong?
The basis you give for $\ker(\mathbf{YA})$ is correct. You're correct that the the dimension of $\mathbf V = \ker(\mathbf{YA})/\ker(\mathbf{A})$ is one, so the basis has only one element. Of course, the elements of $\mathbf V$ are cosets, so that we really want something of the form $\mathbf X + \ker(\mathbf A)$ as the basis element, but I assume the paper is just leaving out the $"+\ker(\mathbf A)"$.
In any event, any element in $\ker(\mathbf{YA})\setminus \ker(\mathbf{A})$ will serve as the required $\mathbf X,$ and at first I could not figure out why they have chosen such a complicated-looking $\mathbf X$ in the paper. By inspection, either of the basis vectors that you give for $\ker(\mathbf{YA})$ will work.
I believe, however that I have figured it out, at least in part. The given vector is orthogonal to $\ker(\mathbf A),$ and this must be what they wanted to achieve. It is not enough, as you suggest in your answer, to pick a vector orthogonal to $\ker(\mathbf A);$ it must also be an element of $\ker(\mathbf{YA}).$
So, the elements of $\ker(\mathbf{YA})$ are of the form $(x/\alpha, y, x/\beta)^T, x,y \in \mathbb R,$ and we must have $$(x/\alpha, y, x/\beta)\cdot (1/\alpha, 1/\varepsilon, 1/\beta) = 0.$$ This gives, $$y = \frac{-\varepsilon x}{\alpha^2+\beta^2}$$ We get the vector from the paper on setting $x=\alpha \beta.$ I have no explanation for why they chose this particular value for $x.$ Some desire for symmetry perhaps?