So I'm trying to write down the maximum function(with a precise mathematical notation) over a set of integers by utilizing the generic maximum function which takes two integers, $max: \mathbb{N} \times \mathbb{N} \to \mathbb{N}$.
So far this is what I've got:
Let $max: 2^{\mathbb{N}} \to \mathbb{N}$ be a function s.t.
$bigmax(S) = max(max(max(max(x_1, x_2), x_3)...), x_n)$ where $x_i \in S$ for $i = [1..n]$.
I would appreciate the feedback on whether this is true or not. I'm especially don't feel comfortable with the three dotted part of the definition.
Depending on what it is exactly you want to do, I don’t think you can or your notation is slightly off.
Finite lists
If you want to describe the maximum for finite lists, $\operatorname{bigmax}$ is not a function $2^\mathbb{N} \to \mathbb{N}$. $2^\mathbb{N}$ is the set of all subsets of $\mathbb{N}$ and first, lists are different from subsets, and second, $2^\mathbb{N}$ also contains infinite subsets, e.g. $\mathbb{N}$ itself or the set of all odd numbers.
Your intended function would then be $$\operatorname{bigmax} : \mathbb{N}^n \to \mathbb{N}$$ if all of your lists have the same length $n$ or maybe $$\operatorname{bigmax} : \bigcup_{n=1}^\infty \mathbb{N}^n \to \mathbb{N}$$ if the length of your lists varies. In either case, I would change your definition to $$\operatorname{bigmax}(x_1,\dotsc,x_n) = \max(\dotsc \max(\max(x_1,x_2), x_3) \dotsc, x_n).$$
Arbitrary subsets
If your intention really is to define a function $2^\mathbb{N} \to \mathbb{N}$, there is a problem: Not all subsets of $\mathbb{N}$ have a maximum, take for example $\mathbb{N}$ itself or the set of odd numbers or the empty set. (In fact, a set of natural numbers has a maximum if and only if it is finite.)
Edit: If you want your function to work on sets, there is a conceptual problem with your definition: Because your function $\max$ only takes two arguments, you have to apply it iteratively and to do that, your elements need to be in some order. Sets are unordered, however. The easiest way would be to simply turn your set into an ascending list, but then the maximum is simply the last element.
Depending on the required precision, maybe the following might work: $$\operatorname{bigmax}(\{x_1,\dotsc,x_n\}) = \max(\dotsc \max(\max(x_1,x_2), x_3) \dotsc, x_n).$$ Note that this way, the elements of your set are implicitly placed in some order (and technically, you would need to prove that this order doesn’t matter).
Your formulation with “where $x_i \in S$ for $i = [1..n]$” is problematic because it isn’t clear that $(x_i)$ is an enumeration of the whole set (and if the set is infinite, it wouldn’t be).