I apologize if this question is on the wrong site, but hopefully I am asking this question on the correct site (and it's not, for example, for DBA professionals).
I am looking to learn a somewhat rigorous approach to relational algebra/calculus used for describing database theory. I already have a decent experience in the real-world with SQL and Database Design, but was wondering if there is a formal take on the mathematics behind it, and if so, a reference that I could use to self-teach.
It's also possible that this isn't a real math discipline worthy of a book/thesis-type discussion, in the same way that basic trigonometry could probably be covered in a few weeks in a Precalculus course and wouldn't require a full-length book to go over. (For reference, I have a background in calculus, linear algebra, and differential equations, so hopefully if it is its own math discipline I'll be suited to understanding it!) Thank you very much for your help and suggestions.
Though it's not a reference in the traditional sense, this free Udacity course covers pretty much exactly what you mentioned (especially in lesson two): relational algebra and calculus.
Do note that I haven't taken the course myself, so I can't comment on the quality of the course, but I hope that it can help you out a bit.
Additionally, for the book that a commenter mentioned, you'll need at least an intermediate understanding of graph theory, computer science and abstract algebra.
The first seven chapters of this graph theory textbook are a good introduction, though a bit wordy at times. Additionally, this book on abstract algebra by Charles C. Pinter is a tad heavy on the lingo from time to time but a very good introduction to most structures you'll encounter in doing database management stuff. Not only that, but the concepts you'll encounter in these books are really good for getting a more "mathematical" insight in a lot of day-to-day things.
Good luck with the self-teaching!