Is there such branch as nonlinear matrix algebra, that researches nonlinear functions with matrix (tensor) valued arguments and outputs where nonlinearities are applied componentwise. Such functions arise in neural networks.
Effectively this math would be the formalization and elaboration of the themes of https://nccd2017.sciencesconf.org/