$$(a)\int_{[0,1]}\beta_i\beta_jd\lambda \\ (b)\int_{[0,1]}\beta_i^2d\lambda$$
The binary notation of a number from $[0,1]$ being: $$\sum_{i=1}^{+\infty}\frac{a_i}{2^i};a_i\in\{0,1\}$$ (I would think) I am guessing this would be done using some sort of geometric sums, but I don't see which. I don't know how to get this one going.
On
Note that $$\beta_i(x)\beta_j(x) = \begin{cases}1 & \beta_i(x)=\beta_j(x)=1 \\ 0 & \mbox{otherwise} \end{cases}$$
That is $\beta_i(x)\beta_j(x)$ is the indicator function for the set $A_{i,j} = \{x\in[0,1]: \beta_i(x)=\beta_j(x)=1\}$, then $$\int_{[0,1]}\beta_i(x)\beta_j(x) d\lambda = \lambda(A_{i,j})$$ where, I am assuming, $\lambda$ is some measure.
Likewise $\beta_i^2 = \beta_i$ is the indicator function for the set $$A_i = \{x\in[0,1]:\beta_i(x)=1\}$$ so $$\int_{[0,1]} \beta_i^2(x)d\lambda = \lambda(A_i).$$
Without knowing more about $\lambda$, I can't say much more.
It would help to just sketch a picture of what the graph of $\beta_k(x)$ looks like.
$\beta_k$ is just the indicator function (a.k.a. characteristic function) of $$(1/2^k,2/2^k] \cup (3/2^k,4/2^k] \cup \cdots \cup ((2^k-1)/2^k, 2^k/2^k].$$
For example, in the case $k=3$, this is $(1/8,2/8] \cup (3/8,4/8] \cup (5/8,6/8] \cup (7/8,8/8]$ and corresponds to the numbers whose binary representations begin with $0.001$, $0.011$, $0.101$, and $0.111$ respectively.
From this you immediately see that the integral (b) is $1/2$, since $\beta_k^2=\beta_k$ is the indicator for $2^{k-1}$ intervals each of length $2^{-k}$.
For (a), if we assume $i<j$, then a similar argument shows the integral is $2^{j-2}/2^j = 1/4$, since $\beta_i \beta_j$ is the indicator for $2^{j-2}$ intervals each of length $2^{-j}$. To see this, consider the $2^{j-2}$ possible outcomes for the first $j$ bits such that bits $i$ and $j$ are both $1$.