I am trying to put the following (Matlab notation) into proper math notation and am having difficulty.
"When X and Y are matrices, ix and iy are such that X(:,ix) and Y(:,iy) are minimally separated"
I have defined X, Y, ix, and iy, the problem is that I don't know how to say X(:,ix) in math notation
If $ix$ is a set of (column) indices, such as $[2,5,4]$, notation $X(:,ix)$ means the submatrix of $X$ obtained by taking all the lines (this is the meaning of the colon symbol), and the columns in the order found in $ix$; example with $ix=[2,5,4]$:
$$X=\pmatrix{1&4&11&3&5&7\\4&2&10&9&0&5} \ \ \rightarrow \ \ X(:,ix)=\pmatrix{4&5&3\\2&0&9}$$
But this description can be converted into this matrix operation:
$$\pmatrix{1&4&11&3&5&7\\4&2&10&9&0&5}\pmatrix{0&0&0\\1&0&0\\0&0&0\\0&0&1\\0&1&0\\0&0&0}=\pmatrix{4&5&3\\2&0&9}.$$
Edit: After a judicious remark of @littleO: Up to my knowledge, there is no commonly accepted mathematical convention for this kind of submatrix. Matlab notational conventions, at least in a certain "applied maths" community, are used as a substitute for non existing "classical maths" conventions.