Given that
$\quad X, Y \in R^{N \times M}$
$\quad D \in R^{N \times N}$
Finding it difficult to proceed differentiation on these matrix equation.
The above form can be rewritten as follows
Equivalent Form
$\quad \min_{\small X} \left ( X-Y \right )\left ( X - Y \right )^{t} + \left ( DX \right )\left ( DX \right )^{t}$
Proceeded Idea
I proceeded with taking the equivalent form of objective function
$\quad f(X) = \left ( Y - X \right )\left ( Y - X \right )^{t} + \left ( DX \right )\left ( DX \right )^{t}$
$\quad f(X) = YY^{t} -YX^{t}- XY^{t} +XX^{t} + DXX^{t}D^{t}$
$\quad f(X) = YY^{t} -\left ( XY^{t} \right )^{t}- XY^{t} +XX^{t} + DXX^{t}D^{t}$
Differentiating the expression
$ = \frac{\mathrm{d} }{\mathrm{d} X} \left ( XY^{t} \right )^{t} - \frac{\mathrm{d} }{\mathrm{d} X} XY^{t} + \frac{\mathrm{d} }{\mathrm{d} X} XX^{t} + \frac{\mathrm{d} }{\mathrm{d} X} DXX^{t}D^{t}$
The expression has terms with three different forms
$\frac{\mathrm{d} }{\mathrm{d} X} XY^{t}$
$\frac{\mathrm{d} }{\mathrm{d} X} XX^{t}$
$\frac{\mathrm{d} }{\mathrm{d} X} DXX^{t}D^{t}$
Is this approach a valid one? Am I missing core idea? Im not sure how to approach these derivatives terms.
It appears from context that your norm is the Frobenius norm, $\|X\|^2=\operatorname{tr}(X^TX)$, which comes from the Frobeinus inner product $\langle X, Y \rangle = \operatorname{tr}(X^TY)$. We can use bilinearity of this inner product to break things apart into pieces that are easier to manage. We are trying to minimize
$$ \langle X,X \rangle-2 \langle X,Y \rangle + \langle Y,Y \rangle + \langle X,D^TDX \rangle.$$
There are various approaches one can take for the derivative of a function of matrices, but one nice approach is the gradient of $f(X)$ satisfies $f(X+h)=f(X)+\langle h,\nabla f(X) \rangle + O(h^2)$. Using this, one can easily show that $\nabla \langle X,X \rangle=2X$, $\nabla \langle X,Y \rangle=Y$, and $\nabla \langle X,AX \rangle=(A+A^T)X.$
Thus, the gradient of the full expression is $2(X-Y)+2D^TDX$. Setting this equal to $0$ yields
$$ X = (I+D^TD)^{-1}Y.$$