For 2x2 matrix for example:
$$K_1 = \pmatrix{K_{xx} \quad K_{xy}\\
K_{yx} \quad K_{yy}}$$
the transformation matrix is:
$$ A = \pmatrix {\cos \theta \qquad \sin \theta\\
\!\!-\sin \theta \qquad \!\!\cos \theta}$$
the above matrix is for rotation in counterclockwise direction, suppose I have a 2x3 matrix for example:
$$K_2 = \pmatrix{K_{xxx} \quad K_{xxy} \quad K_{xyy}\\
K_{yxx} \quad K_{yxy} \quad K_{yyy}}$$ what will be the transformation matrix? and how can I transform the above matrix by, let say $30^\circ$?
Any help is appreciated.
Is your stiffness matrix having some formula like the $ D $ operator defined in https://en.wikipedia.org/wiki/Stiffness_matrix#Practical_assembly_of_the_stiffness_matrix? In that case, it suggests a formula: each column will rotate like a vector. So you will have $ K' = A K $.
Regrettably, I am missing some context and particularly I don't know what a stiffness matrix is. But your answer will be found by understanding what the domain and range of $ K $ are; or in other words, understanding the physical meaning of the matrix. Suppose $ K: V \rightarrow W $, and a rotation $ R $ acts on $ V $ by matrix $ R_v $ and on $ W $ by a matrix $ R_w $, then the transformation for $ K $ is $ R_w K R_v^{-1} $.