Find the analytical solution of the stochastic integral equation $$\tag1 X(t) = -\frac18-\int_0^t\frac14sX(s)ds-\int_0^t\frac{1}{40}X(s)dB_H(s) $$ where $(B_H(t))_{t\ge0}$ is a fractional Brownian motion, i.e. a continuous Gaussian process with $B_H(0)=0$, $\mathbb E[B_H(t)]=0$ $\forall t$, covariance $\mathbb E[B_H(t)B_H(s)] = \frac12(|t|^{2H}+|s|^{2H}-|t-s|^{2H})$ and $H\in(0,1)$.
SOLUTION $$\tag2 X(t) = -\frac18\exp\Big(-\frac{1}{40}B_H(t)-\frac{t^2}{8}-\frac{t^{2H}}{3200}\Big) $$ P.S. I have not to verify that this is the solution, but rather to find it from $(1)$.
I tried to apply the Ito formula for fractional integrals
where $f(s,x) : \mathbb R \times \mathbb R \to \mathbb R$ is a function of $C^{1,2}(\mathbb R \times \mathbb R)$.
Using $f(t,X(t)) = \log X(t)$ I got
$$ \log X(t) = \log X(0)+\int_0^t\frac{1}{X(s)}dX(s)+H\int_0^t-\frac{1}{X^2(s)}s^{2H-1}ds+\underbrace{\int_0^t\frac{\partial\log X(s)}{\partial s}ds}_{=0\ ?} $$
and by substituting $dX(s) = -\frac14sX(s)ds-\frac{1}{40}X(s)dB_H(s)$ (obtained from $(1)$ by differentiating)
$$\begin{align*} \log\frac{X(t)}{X(0)} &= \int_0^t \Big(-\frac14sds-\frac{1}{40}dB_H(s)\Big)+\underbrace{H\int_0^t-\frac{1}{X^2(s)}s^{2H-1}ds}_{=\ ?} \\ &=-\frac{t^2}{8}-\frac{1}{40}B_H(t)+? \end{align*}$$
Then by taking exponential both sides
$$ X(t) = X(0)\exp\Big(-\frac{t^2}{8}-\frac{1}{40}B_H(t)+?\Big) $$
which partially is equal to $(2)$. Moreover I notice that $H\int_0^ts^{2H-1}ds = t^{2H}/2$, which is also contained in $(2)$. But I don't know how to deal with $\frac{1}{X^2(s)}$.
Maybe another way is to assume that the solution is of the form $$ X(t) = a\exp(bB_H(t)+c(t)) $$ but how to find $a, b\in\mathbb R$ and the real function $c$?
You didn't apply Itô's formula correctly. The Itô formula which you stated at the beginning of your calculation applies for expressions of the form $f(t,B_H(t))$ - but in your computation you need to compute the stochastic differential of $f(t,X_t)$. Compare this with the situation in classical stochastic calculus (i.e. for Brownian motion): Itô's formula for Brownian motion looks different than Itô's formula for general Itô processes.
Here is the version of Itô's formula which you need to solve the problem: If $$X_t = \int_0^t b(s) \,ds + \int_0^t \sigma(s) \, dB_H(s)$$
then (assuming everything is well-defined)
\begin{align*} f(t,X_t) -f(0,X_0) &= \int_0^t \partial_s f(s,X_s) \, ds + \int_0^t \partial_x f(s,X_s) \, d\color{red}{X_s} \\ &+ H \int_0^t \partial_x^2 f(s,X_s) \color{red}{\sigma(s)^2} s^{2H-1} \, ds. \tag{1} \end{align*}
Note that for $X_t = B_H(t)$ we get, as a particular case, the Itô formula which you stated.
Applying Itô's formula $(1)$ for $f(t,x) = \log x$, it follows that
\begin{align*} \frac{\log X_t}{\log X_0} &= \int_0^t -\frac{1}{4} s \, ds - \frac{1}{40} \int_0^t dB_H(s) + H \int_0^t - \frac{1}{X_s^2} \color{red}{\left( \frac{1}{40} X_s\right)^2} s^{2H-1} \, ds \\ &= - \frac{t^2}{8} - \frac{1}{40} B_H(t) - \frac{1}{3200} t^{2H}.\end{align*}
(The term in red is the one which is missing in your calculation.) Hence,
$$X_t = X_0 \exp \left(-\frac{t^2}{8} - \frac{1}{40} B_H(t) - \frac{1}{3200} t^{2H} \right).$$