I want to compute the partial derivative of my loss function for $W$ \begin{align} L(W) &= \frac{1}{2}\| X-WZ^{T}\|^{2}_{forb} + \frac{1}{2} \lambda_{w}\| W \| \\ &= \frac{1}{2}(X - WZ^{T})^{T}(X-WZ^{T}) + \lambda_{w}W^{T}W \\ &= \frac{1}{2}\big(X^{T}X - X^{T}WZ^{T} - (WZ^{T})^{T}X + (WZ^{T})WZ^{T} \big) + \lambda_{w}W^{T}W \end{align}
But I must be doing something wrong, as when I derive the last equality with respect to $W$, the dimensions doesn't work any more. $X$ is a $d \times n$ matrix, $W$ $d \times k$ and $Z^{T}$ is $k \times n$
\begin{align} \frac{dL(W)}{\delta w} &= \frac{1}{2}\big(\!-\! X^{T}Z^{T} - (Z^{T})^{T}X + (Z^{T})WZ^{T} + ZWZ^{T}\big) + \frac{2}{2} \lambda_{w}W^{T} \end{align}
The dimension problem can be seen with $X^{T}Z^{T}$ which is $n\times d \times k\times n$. I believe I am missing some derivation rules, and I would be glad if some one would point them out, as I don't find the issue.
Best.
Assuming $forb$ is the Frobenius norm, and that the penalty term is also a squared Frobenius norm, $\lVert W \rVert_{\mathrm{F}}^2$, then your first step is incorrect.
The Frobenius norm squared in the first line is a scalar, while the matrix product in the second line gives an $N$x$N$ matrix. You need to take the trace of that matrix to obtain the Frobenius norm squared:
\begin{align} L(W) &= \frac{1}{2} \lVert X-WZ^{T}\rVert_{\mathrm{F}}^2 + \frac{1}{2} \lambda_{w}\lVert W \rVert_{\mathrm{F}}^2 \\ &= \frac{1}{2}\,\mathrm{tr}\left[(X - WZ^{T})^{T}(X-WZ^{T})\right] + \lambda_{w}\, \mathrm{tr}\left[ W^{T}W \right] . \end{align}
Once the product is expanded, you can use the fact that the trace of the sum is the sum of the traces as well as some useful properties from "Matrix Calculus - Notes on the Derivative of a Trace" by Johannes Traa to compute the desired derivative.