The optimization problem is as follows: \begin{equation} \min_{X \in \mathbb{R}^{n \times n}} ~f(X) + \langle Y, X+X^T \rangle + \|X+X^T\|^2, \end{equation} where $f \colon \mathbb{R}^{n \times n} \to \mathbb{R}$ is convex function and $Y$ is given parameter matrix. The inner product $\langle \cdot, \cdot \rangle$ is Frobenius inner product.
I want to know could I obtain the closed-form solution of this problem. I am stuck for a long time but have no idea at all. Any help will be appreciated.
Your function $f$ is unknown (assume it is differentiable?).
Let us use a colon for the Frobenius product, i.e., $$A:B={\operatorname{Trace}}(A^TB) \equiv \langle A, B\rangle.$$
The cyclic property of the Frobenius product, e.g., $$\eqalign{ A:B &= A^T:B^T &= B:A }$$
Let us define the following with differential. \begin{align} \phi_1 := Y : \left( X + X^T \right) \Rightarrow d\phi_1 = \left( Y + Y^T \right): dX \end{align} and \begin{align} \phi_2 := \left( X + X^T \right) : \left( X + X^T \right) \Rightarrow d\phi_1 = 2\left( X + X^T \right): \left( dX + dX^T \right) = 4\left( X + X^T \right): dX. \end{align}
The composite function can be expressed as \begin{align} \theta = f + \phi_1 + \phi_2. \end{align}
Take the differential of the composite by plugging in the differentials of $\phi_i$, i.e., \begin{align} &d\theta = df + d\phi_1 + d\phi_2 = df + \left( Y + Y^T \right):dX + 4\left( X + X^T \right): dX \end{align}
Then, obtain the gradient and I think you know what to do? \begin{align} 0 \in \frac{\partial \theta}{\partial X} = \frac{\partial f}{\partial X} + \left( Y + Y^T \right) + 4\left( X + X^T \right). \end{align}