I struggle with the following tasks.
To derive $h$ I would use the Quotient rule(?).
When I derive the numerator after $x$ so $$\frac{\partial (y-g(x))^2}{\partial x} = 2 \cdot (y-g(x)) \cdot g'(x)$$
a dimensionality problem occur. $2 \cdot (y-g(x))$ is a (nx1) vector while $g'(x)$ should be a $nx3 matrix (?). How should I handle this?
Same problem with the denominator: $$\frac{\partial (1+(y-g(x))^2)}{\partial x} = 2 \cdot (y-g(x)) \cdot g'(x)$$
I would appreciate any advice
Edit: Here is my new complete solution:
I think the notations of the exam (?) are very messy. If I understand correctly from your indications, $x^2$ means the squared norm (which is a poor notation). Let denote $ \mathbf{u}=\mathbf{g}(\mathbf{x})-\mathbf{y} $.
The scalar-valued function $h$ writes $$ h = \frac{\|\mathbf{u}\|^2}{1+\|\mathbf{u}\|^2} $$ i.e. $$ \|\mathbf{u}\|^2 = \frac{1}{1-h}-1 $$ Taking the differential $$ 2 \mathbf{u}:d\mathbf{u} = \frac{dh}{(1-h)^2} $$ where the colon operator : denotes the inner product between two vectors.
It follows $$ dh =2(1-h)^2 \mathbf{u}:d\mathbf{u} =2(1-h)^2 \mathbf{u}:\mathbf{J}_g\ d\mathbf{x} =\color{red}{2(1-h)^2 \mathbf{J}_g^T \mathbf{u}}:d\mathbf{x} $$ The LHS term is the gradient of $h$ and $\mathbf{J}=\frac{\partial \mathbf{g}}{\partial \mathbf{x}}$ is the Jacobian of $\mathbf{g}$ wrt $\mathbf{x}$.