Wikipedia gives $$\sum_{k=2}^\infty(\zeta(k)-1)=1,\quad\sum_{k=1}^\infty(\zeta(2k)-1)=\frac34,\quad\sum_{k=1}^\infty(\zeta(4k)-1)=\frac78-\frac\pi4\left(\frac{e^{2\pi}+1}{e^{2\pi}-1}\right)$$ from which we can easily find $\sum_{k=1}^\infty(\zeta(2k+1)-1)$ and $\sum_{k=1}^\infty(\zeta(4k+2)-1)$. From here it's natural to ask the following
Question: Is there a known closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-1)$?
Related: Closed form for $\sum\limits_{k=1}^{\infty}\zeta(4k-2)-\zeta(4k)$ and Closed form for $\sum_{k=1}^{\infty} \zeta(2k)-\zeta(2k+1)$.
Progress
Note that we are done once we have a closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-\zeta(4k+3))$. So I tried the same approach as in one of the questions above and this is what I got:
We have $$\zeta(4k+1)-\zeta(4k+3)=\sum_{n\geq2}^\infty\left(1-\frac1{n^2}\right)\frac1{n^{4k+1}}$$
Hence
$$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k+3)=\sum_{n\geq2}\left(1-\frac1{n^2}\right)\sum_{k\geq1}\frac1{n^{4k+1}}=\sum_{n\geq2}\left(1-\frac1{n^2}\right)\frac1{n^5}\frac1{1-\frac1{n^4}}\\=\sum_{n\geq2}\frac1{n^3+n^5}.$$
In the same way I get $$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k)=-\sum_{n\geq2}\frac1{n+n^2+n^3+n^4}$$ and $$\sum_{k\geq1}\zeta(4k+1)-\zeta(4k+2)=\sum_{n\geq2}\frac1{n^2+n^3+n^4+n^5}.$$ It suffices (in fact it is equally hard) to evaluate any of these series.
I don't know if this will be useful, but thought that is might help. So, here we go ...
We can write the sum of interest as
$$\begin{align} \sum_{k=1}^{\infty}\left(\zeta(4k+1)-1\right)&=\sum_{k=1}^{\infty}\sum_{n=2}^{\infty}\frac{1}{n^{4k+1}} \tag 1\\\\ &=\sum_{n=2}^{\infty}\left(\frac{n}{2(n^2+1)}+\frac{1}{4(n-1)}+\frac{1}{4(n+1)}-\frac{1}{n}\right) \tag 2\\\\ &=\sum_{n=2}^{\infty}\left(\frac{n}{2(n^2+1)}-\frac{1}{2n}\right) \tag 3\\\\ &+\sum_{n=2}^{\infty}\left(\frac{1}{4(n-1)}-\frac{1}{4n}\right) \\\\ &+\sum_{n=2}^{\infty}\left(\frac{1}{4(n+1)}-\frac{1}{4n}\right) \\\\ &=\frac18-\frac12\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}\tag 4 \end{align}$$
In arriving at $(1)$, we used the series definition of the Riemann-Zeta function.
In going from $(1)$ to $(2)$, we changed the order of the series, summed the interior geometric series, and expanded in partial fractions.
In going from $(2)$ to $(3)$, we merely split the sum of three convergent series.
In arriving at $(4)$, we noted that the latter two series in $(3)$ are telescoping series whose sum is $\frac18$.
Now, this last series can be written in terms of the Digamma Function as
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=-1+\gamma+\frac12\psi^0(2-i)+\frac12\psi^0(2+i)\approx 0.17186598552401 \tag 5$$
Therefore, we have
$$\sum_{k=1}^{\infty}\left(\zeta(4k+1)-1\right)=\frac58 -\frac12 \gamma-\frac14\psi^0(2-i)-\frac14\psi^0(2+i)\approx 0.039067007237995$$
NOTE:
Here we show that the series in $(5)$ is indeed given by
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=-1+\gamma+\frac12\psi^0(2-i)+\frac12\psi^0(2+i)$$
Expanding in partial fractions yields
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=\frac12\sum_{n=2}^{\infty}\left(\frac1n-\frac{1}{n+i}\right)+\frac12\sum_{n=2}^{\infty}\left(\frac1n-\frac{1}{n-i}\right) \tag 6$$
Next, using the series definition of the digamma function, we find
$$\begin{align} \psi^0(z+1)&=-\gamma+\sum_{n=1}^{\infty}\frac{z}{n(n+z)}\\\\ &=-\gamma+\sum_{n=1}^{\infty}\left(\frac1n-\frac{1}{n+z}\right)\\\\ &-\gamma+\sum_{n=1}^{\infty}\left(\frac1n-\frac{1}{n+1}+\frac{1}{n+1}-\frac{1}{n+1+(z-1)}\right)\\\\ &=-\gamma+1+\sum_{n=2}^{\infty}\left(\frac{1}{n}-\frac{1}{n+(z-1)}\right) \tag 7\\\\ \end{align}$$
Using $(7)$ in $(6)$, we can write
$$\begin{align} \sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}&=\frac12 \left(-2\gamma+2+\psi^0(2+i)+\psi^0(2-i)\right)\\\\ &=-1+\gamma +\frac12\psi^0(2+i)+\frac12\psi^0(2-i) \end{align}$$
as was to be shown!