I'm trying to tackle this problem of showing that the Lorentz group $$O(1,3)=\{M\in GL(4, \mathbb{R})\mid M\eta M^t=\eta\},$$ is non-compact. In order to do so I need to show that it is either not bounded or not closed.
For real numbers the idea of bounded sets make perfect sense. From Wikipedia "A set $S$ of real numbers is called bounded from above if there exists some real number k (not necessarily in S) such that $k\geq s$ for all $s\in S $. The term bounded from below are similarly defined".
What is now my question is, how this notion of upper and lower bounds apply to groups. Is it as simple as saying that there exists a real number $k$, such that $k \geq m$ for all $m \in M$?
Short answer: Each entry of the matrix must be bounded [see page 16. of Brian C Hall's book on Lie groups]. In particular, having bounded determinant is not enough (there is a counter-example on the same page).
Long answer: Note that the result that a subset is compact iff it is closed and bounded is a special property of subsets of $\mathbb R^n$, which is the content of the Heine-Borel theorem. It is not generally true for a topological vector space (TVS).
Now, matrix Lie groups are at the intersection of topology and groups, so when we ask whether a matrix Lie group has a certain property, we want to first think about whether we're looking at it as a group or as a TVS. Boundedness is a property of a TVS and its definition is given here. The 'boundedness' you're thinking of is the special case of boundedness of intervals in $\mathbb R^1$, which is a TVS.
On page 16 of Brian C. Hall's book on Lie groups, he identifies (as in he considers these to be identical, in some sense) a matrix in $GL(n;\mathbb C)$ with the corresponding vector in $\mathbb R^{2n^2}$, where he has 'stacked' the real and imaginary parts of each entry of the matrix into a vector. Now we can just ask whether the set of matrices, interpreted as points in $\mathbb R^{2n^2}$, constitute a bounded subset of $\mathbb R^{2n^2}$. No longer do we need to talk about boundedness of groups, Lie groups, or matrices, just vectors in a Euclidean space. This is what I mean by 'viewing the Lie group as a TVS' - to ascertain whether a matrix Lie group is bounded, we need not concern ourselves with its qualities as a group.
Thereafter, we have that $A \subseteq \mathbb R^{2n^2}$ is bounded if $\exists c \in \mathbb R$ such that for all $x\in A$, $\| x\|_2 < c$. This follows from the definition of bounded sets linked above. Finally, we can use the equivalence of norms (i.e., vectors are big/small in one norm iff they are big/small in another norm) to replace this condition with: $\|x\|_{\infty}<\tilde c$, which is saying that the absolute value of each entry of the matrix is bounded from above by some constant.