(note regarding notation for function application: I sometimes write $f(x)$ as $f.x$, to simplify brackets)
I begin by defining the key pieces of the random variable definition, in a way that is more detailed than necessary, for the sake of pedagogy.
A "random variable" isn't random or a variable. It is a surjective function between a sample space $S$ and a set of real values in a set $V \subset \mathbb{R}$. Let $X$ be some such function. Even though $X$ is surjective and thus does not necessarily have an inverse, we will still be interested in the co-function: $\tilde{X}: V \rightarrow A \subset S$, which maps a value in $V$ to some subset of elements in $S$ in this way: $$\tilde{X}.v = \bigcup\{A \subset S: X.A = v\}$$
Recall that a probability function $P$ over $S$ maps subsets of $S$ to a real numbers in $U$, where $U$ is such that $U \subset [0, 1]$, and $\sum U = 1$. This means that $P.\tilde{X}: V \rightarrow A \subset S \rightarrow U$ is a probability function on the set $V$, governed by the structure of $P$ on $S$.
The function $p_X = P.\tilde{X}$ is called the "mass" function if $S$ is finite in size.
Let $S = \{A, B, C\}$, and let $V = \{1, 2\}$. Let $P$ be a probability function over $S$ such that $P(A) = P(B) = P(C) = 1/3$.
Let $X$ be a random variable such that $X: R \subseteq S \rightarrow V$, such that: $X.\{A, B\} = 1 = X.\{B, C\}$ and $X.\{A, C\} = 2$. Let $p_X$ be the probability mass function for the RV $X$.
How will $p_X$ be defined? Note that $p_X = P.\tilde{X}$. So we first need to construct $\tilde{X}$. There is two values in $V$: \begin{align} &\tilde{X}.1 = \bigcup \{E \subset S : X.E = 1\} = \{A, B\} \cup \{B, C\} = \{A, B, C\} \\ &\tilde{X}.2 = \bigcup \{E \subset S : X.E = 2\} = \{A, C\} \end{align} So: \begin{align} &p_{X}.1 = P.\tilde{X}.1 = P.\{A, B, C\} = 1 \\ &p_{X}.2 = P.\tilde{X}.2 = P.\{A, C\} = 2/3 \end{align} What is the sum of $p_{X}$ over all the values it can take? \begin{align} \sum_{y \in V} p_{X}.y = p_{X}.1 + p_{X}.2 = 1 + 2/3 = 5/3 \end{align}
Shouldn't the sum of $p_X$ over all its values sum to $1$ though? What am I doing wrong?
A closer reading of my textbook suggests the following:
1) a random variable is simply any function from a sample space, to a subset of the real numbers.
2) a random variable is a discrete random variable if for a set of finite or countably infinite values $\{k_1, k_2, \dots\}$, where $\{k_i\} \subset \mathbb{R}$, we have that $\sum_{i} P.\tilde{X}.k_i = 1$.
What is a random variable for which an appropriate $\{k_i\}$ satisfying point 2 has not been defined called? This is what I constructed in the first part of the question.
An easy way to fix the problem would be to renormalize $P.\tilde{X}.v$ by what it sums to be:
$$ P.\tilde{X}.1 = 1/(5/3) = 3/5, \; P.\tilde{X}.2 = 2/3/(5/3) = 2/5$$
What would such renormalization be called?
your mistake is your definition of X. As it is a function from S, you need to set X for each element of S and not for subsets of S. Ie you need to define $X.A, X.B, X.C$ (and not $X.\{A,C\}$, $X.\{A,B\}$ or $X.\{B,C\}$) so the sum of probabilities will be consistent.