Where do I go from Linear algebra past Calc III to try to learn complex physics (relativity and quantum group theory)?

Question

I'm mainly a programmer, but I have a love for Mathematics that's been, well, insatiable. I've had my eye on learning Quantum Groups and Relativity, but I want to stay in something I can do with Computer Graphics. What do I need to know?

I have got a solid grasp of:

1. Calc I,

2. Clac II

3. Some Calc III (basically 3D+ applications of single and multidimensional calculus).

4. Linear Algebra (for graphics programming)

I have a decent grasp of:

1. Abstract Algebra,

2. Set Theory

3. Binary Lattices (mostly from computer mathematics).

I do have a book on finite groups that is really interesting. However I have to admit, that other than knowing that in hyperbolic geometry sinx - cosx = 1 is nearly the extent of that subject.

So, what I think I'm asking is how to go from where I'm at, to get to the calculus of manifolds, riemannian geometry, so on.

EDIT: I think what I'm looking to do is mainly Theoretical/Mathematical Physics, specifically things that I can make into awesome graphics programs.