The dot product differentiation rule is $(\vec f(t) \cdot \vec g(t))' = \vec f\ '(t) \cdot \vec g(t) + \vec f(t) \cdot \vec g\ '(t)$, which simplifies to $$(\vec f(t) \cdot \vec f(t))' = 2(\vec f\ '(t) \cdot \vec f(t))$$ when we plug a single vector in for both $\vec f(t)$ and $\vec g(t)$. For example, we find the same answer of $36t^3 + 2t + 4$ whether we plug $\begin{bmatrix}t + 2 \\ 3t^2\end{bmatrix}$ into the LHS or the RHS of the simplified equation. However, when the vectors are functions, I'm having trouble applying this to inner product differentiation.
Let's use the inner product $\int_{-1}^1 f(t)g(t)\ dt$ and the vector $2t^3 + 10$. It seems to me that we should find $(\int_{-1}^1 (2t^3 + 10)(2t^3 + 10)\ dt)' = 2\int_{-1}^1 (6t^2)(2t^3 + 10)\ dt$, by matching the form of the dot product analog. Unfortunately, this LHS must be $0$ (independent of the choice of vector), since the definite integral it contains must return a constant. The RHS is more interesting and works out to $80$, which is more reasonable. Why does this calculation fail, and what is the correct way to migrate the dot product example into an inner product one?
The RHS of your integral example doesn't make sense. You have tried to use $$\frac{d}{dt}\int_a^b f(t) \,dt=\int_a^b \frac{df}{dt} \,dt$$This is simply not true. What is possible is $$\frac{d}{dx}\int_a^b f(x,t) \,dt=\int_a^b \frac{\partial}{\partial x} f(x,t)\,dt$$This is how your rule would apply in the functional case - you'd need functions of multiple variables, and these extra variables are what you differentiate with respect to.
If you did not have this, then $\langle f,g\rangle$ is just a constant, analogous to the dot product between two constant vectors - there is nothing to differentiate with respect to. In the vector case, we were able to differentiate them by promoting them to functions of an extra variable than they originally had - from $0$ to $1$. We need to do something similar for functions, but then to promote them from single-variable functions to multi-variable functions.