I am reading Carroll's book on General Relativity, but this is much more a math question than a physics question. We can approximate a metric tensor on a curved manifold by writing $g_{\mu\nu} = \eta_{\mu\nu} + h_{\mu\nu}$ where $g_{\mu \nu}$ is the metric tensor on the curved manifold and $\eta_{\mu \nu}$ is the metric tensor on Minkowski space and $h_{\mu\nu}$ is some small perturbation. He says that because $g^{\mu\nu}g_{\nu\sigma} = \delta^{\mu}_{\sigma}$ then, to first order in $h$, we have $g^{\mu\nu} = \eta^{\mu\nu} - h^{\mu\nu}$ where $h^{\mu\nu} = \eta^{\mu \rho} \eta^{\nu \sigma}h_{\rho \sigma}$. I don't see how this is the form of $g^{\mu \nu}$ to first order in $h$ though. Suppose $g$ does indeed have that form, then \begin{align*} g^{\mu \nu}g_{\nu \sigma} &= (\eta^{\mu \nu} - h^{\mu \nu})(\eta_{\nu \sigma} + h_{\nu \sigma}) \\ &= \eta^{\mu \nu} \eta_{\nu \sigma} + \eta^{\mu \nu} h_{\nu \sigma} - h^{\mu \nu} \eta_{\nu \sigma} - h^{\mu \nu}h_{\nu \sigma} \end{align*}
We want this be equal to $\eta^{\mu \nu} \eta_{\nu \sigma}$, meaning all those other terms have to be quadratic in $h$, but none of them appear to be?
An analogy can be made with $(1+x)^{-1}$, which has expansion $$\frac{1}{1+x} = 1 - x + x^2 - x^3 + \mathcal{O}(x^4).$$ This suggests that if $g^{\mu\nu}=\eta^{\mu\nu} + h^{\mu\nu}$, then we should have the inverse metric tensor $$g_{\mu\nu} = \eta_{\mu\nu}-h_{\mu\nu} + \eta^{\sigma\rho}h_{\mu\sigma}h_{\nu\rho} + \mathcal{O}(h^3).$$ This is the quick and dirty way to remember this result. The formal way to see this is to define the operator $$g^{\mu}_\nu \equiv \eta_{\mu\sigma}g^{\sigma\nu} = \delta^{\mu}_\nu + h^\mu_\nu.$$ This is a $(1,1)$ tensor, which can be regarded as a linear operator $g:V\rightarrow V$. What we are interested in is a series expansion for $g^{-1}$, which is given by the Neumann series. The Neumann series expansion gives us $$g^{-1} = I - h + h^2 - h^3 + \mathcal{O}(h^4),$$ where I've dropped index notation since the contractions are obvious. But of course $g^{-1}$ is just $$\left(g^{-1}\right)^\mu_\nu = \eta^{\mu\sigma}g_{\sigma\mu}.$$ So we have $$g_{\mu\nu} = \eta_{\mu\sigma}\left(g^{-1}\right)^{\sigma}_\nu = \eta_{\mu\nu} - h_{\mu\nu} + \eta_{\mu\sigma} h^\sigma_\rho h^\rho_\nu + \mathcal{O}(h^3),$$ exactly as claimed. Note that your expression for $g^{\mu\nu}g_{\nu\sigma}$ is quadratic in $h$, since $$\eta^{\mu\nu}h_{\nu\sigma} = h^\mu_\sigma = h^{\mu\nu}\eta_{\nu\sigma},$$ so the two linear terms cancel.