I have three $3D$ vectors ($3\times 1$ matrix) $\vec a$, $\vec b$ and $\vec t$ and also a $3\times 3$ matrix $M$. If $f$ is define as below: $$f=\frac{\left\lVert M\vec a \times (\vec b-\vec t) \right\rVert_2}{M\vec a.(\vec b-\vec t)}$$
what is the the partial derivatives of $f$ wrt $t$ and $M$, i.e. $\frac{\partial f}{\partial t} $ and $\frac{\partial f}{\partial M} $?
Note:
1) $\left\lVert ... \right\rVert_2$ stands for norm-2 (magnitude of the vector)
2) "." stands for inner (dot) product
3) "$\times$" stands for cross product
4) $M\vec a$ is a conventional matrix product
Consider this scalar function of two vectors $(y,z)$, where the first vector $(y)$ is a constant. $$\eqalign{ \phi &= \frac{\|y\times z\|^2}{(y:z)^2} \cr &= \frac{(z:z)(y:y)-(z:y)^2}{(z:y)^2} \cr &= \frac{(z:z)(y:y)}{(z:y)^2} - 1 \cr \cr d\phi &= \frac{2(z:dz)(y:y)}{(z:y)^2} - \frac{2(z:z)(y:y)(y:dz)}{(z:y)^3} \cr &= \frac{2(y:y)}{(z:y)^3}\,\Big((z:y)z-(z:z)y\Big):dz \cr }$$ Note how this function is related to your function $$\eqalign{ \phi &= f^2 &\implies d\phi=2f\,df \cr }$$ Now substitute $$\eqalign{ y &= Ma \cr z &= (b-t) &\implies dz=-dt \cr }$$ to obtain $$\eqalign{ df &= \frac{d\phi}{2f} \,\,= \frac{(y:y)}{(z:y)^3f}\,\Big((z:y)z-(z:z)y\Big):(-dt) \cr \frac{\partial f}{\partial t} &= \frac{(y:y)}{(z:y)^3f}\,\Big((z:z)y-(z:y)z\Big) \cr\cr }$$ A second set of substitutions $$\eqalign{ y &= (b-t) \cr z &= Ma &\implies dz=dM\,a\cr }$$ yields $$\eqalign{ df &= \frac{(y:y)}{(z:y)^3f}\,\Big((z:y)z-(z:z)y\Big):dM\,a \cr &= \frac{(y:y)}{(z:y)^3f}\,\Big((z:y)z-(z:z)y\Big)a^T:dM \cr \frac{\partial f}{\partial M} &= \frac{(y:y)}{(z:y)^3f}\,\Big((z:y)z-(z:z)y\Big)a^T \cr\cr }$$ Throughout the derivation, a colon was used to represent the inner product, i.e. $$\eqalign{ a:b &= a\cdot b = a^Tb \cr A:B &= {\rm tr}(A^TB) \cr }$$ since it is easier to type, and it applies to matrices as well as vectors.