I am looking at the derivation of the Einstein field equations as the Euler-Lagrange equations of the Hilbert functional. To do this one starts with a variation $$ g(t) = g+th $$ of the metric, where $h$ is a symmetric 2-covariant tensor. For small $t$, $g(t)$ will be invertible (if we interpret it as a matrix), so it makes sense to consider the components $g(t)^{ij}$ of the inverse. We have $$ 0 = \frac{d}{dt}\Big|_0 (g(t)_{ij} \, g(t)^{jk}) = h_{ij} g^{jk} + g_{ij}h^{jk} \quad\Rightarrow \quad h^{lk} = - g^{il} g^{jk} h_{ij}. $$ My question: What exactly do the coefficients $h^{lk}$ represent?
I always thought that if one has a, say, 1-covariant tensor $A = A_i dx^i$, then $A^i$ denote the components of the 1-contravariant tensor $A^\#$ (cf. musical isomorphisms), given by $A^i = g^{ij}A_j$. But in the formula for $h^{lk}$ above we also have a minus sign in front, so as far as I can see the coefficients $h^{lk}$ are not obtained by raising the indices of $h_{lk}$. That being said, the metric is not really fixed in this case so the musical isomorphisms aren't either, which may be the source of confusion for me.
Your mistake is that $$h^{jk} \neq \frac{d}{dt}\bigg|_{t=0} g(t)^{jk}.$$In fact, $h^{jk}$ is obtained from raising both indices of $h_{jk}$. What you have shown is that $$\frac{d}{dt}\bigg|_{t=0} g(t)^{\ell k} = -g^{i\ell}g^{jk}h_{ij} = - h^{\ell k}.$$The philosophy behind it is: you know that the initial velocity of the curve $t\mapsto g(t)$ is $h$, and you want to compute the initial velocity of other geometric objects associated to $g(t)$ in terms of $h$. The initial velocity of the curve of inverse coefficients is $-h$ (up to musical isomorphisms).
Dealing with coefficients, you're doing computations on matrix Lie groups. There, the formula for the derivative of the inversion $\iota(A)=A^{-1}$ is given by $d\iota_A(H) = - A^{-1}HA^{-1}$, so everything is fine (compare with $(1/x)'=-1/x^2$ and make it non-commutative; each $A^{-1}$ corresponds to raising one index).