Let $f$ denote a contact force, and $\hat{n}$ the unit normal of a surface at the point where the force is applied. The contact force can be split into tangential and normal components as follows: $$ f = f_{||} + f_{\perp}. $$
I am looking for a matrix $A$ which gives the tangential component $f_{||}$ in the form of $f_{||} = Af$. For example, for $f = [3,4,5]$ and $\hat{n} = [0,0,1]$, the solution is $$ A = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{bmatrix}, $$ which would give the tangential component $f_{||} = Af = [3,4,0]$.
My question is: without rotating $f$, but given the rotation matrix that rotates the unit vector [0,0,1] into the surface normal $\hat{n}$, how can I get $A$?
Thank you in advance!
What you are asking for is given $\hat n$ to find a matrix that "kills" $\hat n$, so you need $A\hat n=0$. We can do this by considering a change of basis $\{\hat x,\hat y,\hat z\}$ to $\{\hat x,\hat y,\hat n\}$, then multiplying by $$\begin{bmatrix}1&0&0\\0&1&0\\0&0&0\end{bmatrix}$$ to eliminate $\hat n$, then changing back to the original basis. The whole process is represented by the matrix $$P\begin{bmatrix}1&0&0\\0&1&0\\0&0&0\end{bmatrix}P^{-1}$$ where $$P=\begin{bmatrix}1&0&n_1\\0&1&n_2\\0&0&n_3\end{bmatrix}.$$
Furthermore it also has to preserve the tangential component, so you need $A\hat t=\hat t$ where $\hat t$ is the unit vector in the tangential direction. To do this is easy: just multiply the matrix above by a constant to make sure it works. I'll leave that to you.