I was looking at back propagating a gradient through a computational graph, and all makes sense aside from when a node has multiple outputs. Take the following function:
$$f(x) = (x+3)(x+2)$$
Which obviously has the derivative:
$$f'(x) = 2x+5$$
Now, take the following segment of a computational graph:
The numbers above a line represent the value on a forward pass, the numbers beneath it represent the back propagated gradient.
When moving the gradient backwards through the $f(x)$ node, I took the sum of $0.34$ and $-0.2$, then multiplied this sum by the derivative of $f(x)$, with an input of $2$ (taken from the forward pass). So: $(0.34-0.2)\times f'(2) = 1.26$. I understand I had to multiply the $f'(x)$ (following the chain rule), but I do not understand why I had to sum the $0.34$ and $-0.2$. I only did so, because I know that's what I'm meant to do.
Any help is greatly appreciated.
In a computation like $$ f(x)=a(u(x))\cdot b(u(x)) $$ you can derive the gradient rules by starting with the universal relation of automatic/algorithmic differentiation between gradients $\bar f, \bar b,..$ and directional derivatives $\dot x,\dot u,..$ which are $$ \bar f\dot f = \bar a\dot a+\bar b\dot b=\bar u\dot u=\bar x\dot x $$ Now you can insert the chain rule and evaluate it in both directions of the identity chain. $\dot f=\dot ab+a\dot b$, $\dot a = a'(u)\dot u$, $\dot b = b'(u)\dot u$ so that $$ \bar f\dot ab+\bar fa\dot b= \bar a\dot a+\bar b\dot b $$ which has to be true for all direction vectors implying $\bar a=\bar f b$, $\bar b=\bar f a$ and $$ \bar aa'(u)\dot u+\bar bb'(u)\dot u=\bar u\dot u $$ implying $\bar u = \bar aa'(u)+\bar bb'(u)$.