What Field(s) Cover Calculus with Matrices and High Dimensional Geometry?

Question

I am taking a course in machine learning and have found that the linear algebra and multivariable calculus from my engineering degree only take me part way in understanding some derivations.

One specific example is differentiating things related to matrices (like differentiating wrt a matrix whose determinant appears in the function...) But this is by no means the only fuzzy bit. I have done just enough geometry, linear algebra and low dimensional calculus to have a notion that these things somehow extend into things involving matrices and things involving high dimensional spaces.

I've tried looking on places like wikipedia's topic listing in mathematics, the page "Categories within Mathematics" at arxiv.org, and undergraduate mathematics curricula however I don't think I even know enough to know whether I'm looking at what I'm looking for... if that makes sense....

Also, I've found some compilations of matrix derivatives but a) it's disconnected from any context and b) seems like a cookbook solution and so these haven't been satisfying.

So... how does what I'm saying here map to topics in mathematics?

If I can cheat and ask a "sub question"... whatever these topics end up being, what are some typical paths people take to get from fairly applied linear algebra and calculus to these "advanced" topics? Items on such a path could be books, names of courses, whatever... I just need some guidance towards context and prerequisites.

Answer 1

I am an engineer, and need to use the "practical" side of these things on a daily basis. Assuming you have a strong grasp of 3D calculus and linear algebra, you probably want to start with a bit of each of the following:

• Multilinear algebra. This is where, e.g. tensors appear. Key topics are:

  • Multilinear forms (especially bilinear forms, and how to represent them with matrices).
  • The tensor product of vector spaces (if you're feeling brave).
  • The exterior algebra of a vector space (if you're feeling extra brave). Understand the coordinate-free definition of the determinant.

• Calculus on (finite-dimensional, real) Banach spaces, generalizing (some of) what you learned in multivariable calc. Key topics are:

  • Directional (Gateaux) derivatives. Understand the isomorphism between vectors and directional derivatives along vectors ($v \sim (f \mapsto \nabla_vf$)).
  • Total (Frechet) derivative. The usual $f'$ is a special case of this, although you'll need to start thinking of $f'$ as a linear map $v \mapsto f'(a)v$ rather than a "number".
  • The Jacobian matrix as the matrix representing the full derivative with respect to standard coordinates.
  • Understand higher-order full derivatives as multilinear forms, at least up to the second derivative. For scalar valued functions, understand how the Hessian matrix is the matrix representation of the second (full) derivative.
  • Integration of vector-valued functions on Euclidean space. I'm not sure there's really more to learn here than what you learned in multivariable calc.
  • Integration of differential forms on chains, if you're feeling extra brave. Understanding the exterior algebra is a prerequisite to this.

These should at least get you more comfortable working with calculus on more sophisticated objects than functions from $\mathbb R^n \to \mathbb R$. Once you're there, you may be interested in doing calculus on non-linear spaces... e.g. how do you differentiate a function on a matrix group, or on the sphere? At that point you may want to dip your toes into differential geometry... Key topics would include pushforwards of maps between manifolds, integration of differential forms, affine connections, Lie groups and the exponential map, etc... But that's probably a ways away for now :)