My book "Cryptography Theory and Practice" by Douglas R. Stinson states that the formulas for encrypting and decrypting on page 19 are follow:
For a key $K$, we define
$$e_K(x)=xK$$
and
$$d_K(y)=yK^{-1}$$
Where all operations are performed in $\mathbb{z}_{26}$
but in all sources like wikipedia, journal, etc state that the formula is:
$$e_K(x)=Kx$$
and
$$d_K(y)=K^{-1}y$$
Which one is correct? $K$ is a matrix, and i know matrices aren't commutative. Thus, those formulas are different.
I've tried encrypted and decrypted with the first formula, i got different result. But when i tried the second formula, i got the same result.
Is that mean the book is wrong or what?
Please help me to clear this. Thanks.
Your book has adopted row vector convention. That is your $x$ is a row vector.
Wikipedia has adopted column vector convention. That is $x$ is a column vector.
To change one convention to the other convention, you just have to transpose everything, note that $$(xK)^T=K^Tx^T$$ where $x$ is a row vector here.
You have to fix and state your convention clearly.