I have always interpreted "pair, triple, quadruple" as "sets containing two, three and four elements". I have never checked this assumption.
Consider the following examples:
The pair $(V, \|\cdot\|)$ is a normed space. Another example is that of a graph with vertices and edges $(V, E)$.
The triple (triplet) $(S, (u_i)_{i \in S}, (a_i)_{i \in S})$ is a game.
The quadruple $(V, F, +, \times)$ is a vector space.
Are these things sets? For example, I have seen people defining "sub-graph", "sub-game", etc. which essentially implied to me that these things are sets.
It occurred to me that it might be strange to think of them as sets, because the set elements are vastly different from each other, e.g. the vector space example, or a digraph, where we insert an additional operation $o$ that specifies the orientation.
What sort of mathematical structures are these objects? Is there anyway to define operations on these objects? What are all the operations that can be defined on these objects?
Yes and no…
A set is an unordered collection of elements. So when a definition says that "The ordered pair $(V,\|\cdot\|)$ is a normed space, where…", the word "ordered" says that this is not a set. Moreover, the word "ordered" would often be omitted, because conventionally "pairs", "triples", …, "tuples" have the meaning of "ordered pairs", "ordered triples", …, "ordered tuples".
That being said, as already mentioned in another (really good!) answer, pairs and longer tuples can be modeled as sets, but that's not important here. People working with normed spaces defined as pairs $(V,\|\cdot\|)$ hardly ever think of them as the set $\{\{V\},\{V,\|\cdot\|\}\}$.
Note that in this example (and similarly in all the other examples that you mentioned), the two elements of the pair $(V,\|\cdot\|)$ are very different things! $V$ itself is a set of elements of that space, while $\|\cdot\|$ is a function $\|\cdot\|\colon V\to\mathbb{R}$.
Thanks to them being different things, we probably don't really need the order in "$(V,\|\cdot\|)$", as it would be easy enough to interpret "$(\|\cdot\|,V)$" as the same thing without any confusion. My guess (and I could be wrong) is that the tradition of setting up these definition as (ordered!) pairs or tuples is not a necessity but rather a matter of great convenience. For once, if we're defining something in math, it should be defined rigorously and unambiguously. Even if there are some random and not so important choices to be made, we should make them anyway, to ensure mutual understanding. Second, defining "$(\|\cdot\|,V)$" would be terribly inconvenient, because the way we read it we would naturally want to define $\|\cdot\|$ first — but we can't: we have to use $V$ in its description, but it hasn't been defined yet as it comes second in our pair. Isn't it more convenient if our notation reflects what we do? So let's put $V$ first!
That's a very common abuse of language and notation that you'll have to get used to. In all these structures there's the underlying set, such as $V$, and additional items that describe a certain structure (the norm function for a normed space; the set of edges for a graph; the base field and the two operations for a vector space; etc.). It's very common to use the same name for the underlying set $V$ only when the context is clear. And since $V$ is a set, it has subsets, called "subspaces", "subgames", etc.
Here's an example. Once again, a normed space is a pair $(V,\|\cdot\|)$, where $\|\cdot\|\colon V\to\mathbb{R}$ is a function (subject to certain properties, not important for this discussion). A "normed subspace" naturally should be a normed space in its own right. So the correct definition is that a normed subspace of a normed space $(V,\|\cdot\|)$ is a normed space $(U,\|\cdot\|')$, where $U\subseteq V$ and $\|\cdot\|'\colon U\to\mathbb{R}$ is defined by restricting the domain of $\|\cdot\|$ to $U$. But since most of this is clear enough, it's quite standard to say "Let $U$ be a subspace of $V$" — which is abuse of language and notation, but it is an accepted shorthand to refer to the technically correct definition of being a "normed subspace".