So i've implemented a convolutional neural network in C++ and I'm playing around with loss functions and gradient calculations. The network outputs the f(a), f(p) and f(n) of the last layer, which is a vector of 10 feature embeddings. They are all L2 normalized before they are used in the loss or gradient calculation.
I've tried different methods for loss and gradients and currently i use the one from this https://arxiv.org/abs/2201.11307 paper. The gradients are weighted accordingly as listed down below.
Since, I've stumbled upon a gradient update method that makes the updating gradient of the negative move orthogonally to the anchor and positive movement. It's described in this paper. https://openaccess.thecvf.com/content_CVPR_2020/papers/Mohan_Moving_in_the_Right_Direction_A_Regularization_for_Deep_Metric_CVPR_2020_paper.pdf
The loss and gradient updating is shown below
Gradient update with orthogonal movement
The following paper reviews the different approaches to loss and gradient update functions and suggests that the cosine space is better to allow more difference between anchor and positive pairs: https://arxiv.org/abs/2201.11307.
The above gradient updating is defined in terms of euclidian space, I'm trying to wrap my head around how to implement the orthogonal movement of the negative gradient. I might be to bad at math to ever do it, can anyone help me out?