Demonstrating Density Using the Stone-Weierstrass Theorem

Question

In exploring the dense subsets of $C\left(I_n\right)$, I've been particularly focused on the application of a foundational result:

$\textbf{Stone-Weierstrass Theorem}$: This theorem posits that for any compact set in $\mathbb{R}^n$ and an algebra $\mathcal{A}$ of continuous real-valued functions on this set, if $\mathcal{A}$ separates the points and includes constant functions, then $\mathcal{A}$ is dense in $C(K)$.

Our interest lies in the functions of the form: $$ G(x)=\sum_{k=1}^M \beta_k \prod_{j=1}^{N_k} \varphi\left(w_{j k}^T x+\theta_{j k}\right) (1) $$ where $w_{j k} \in \mathbb{R}^n, \beta_j, \theta_{j k} \in \mathbb{R}$, and $x \in I_n$, under the $\textbf{valid property}$ that $\textbf{they form a dense subset in $C\left(I_n\right)$}$ using a nonconstant activation function $\varphi$.

My approach for proving it depends on identifying the set $\mathcal{U}$, given by the finite sums of products of the specified type, as an algebra of real continuous functions on $I_n$ that satisfies the conditions of the theorem: separation of points and inclusion of constant functions.

Despite understanding the theorem's prerequisites, I'm seeking a more rigorous mathematical justification to firmly establish $\mathcal{U}$ 's adherence to these conditions, thus proving its density in $C\left(I_n\right)$. I would greatly appreciate any guidance or detailed mathematical rigor that could illuminate this process."

Answer 1

Your approach to applying the Stone-Weierstrass Theorem through the lens of neural networklike functions is indeed intriguing. One aspect that might further illuminate your proof is considering the richness of the space generated by your function $G(x)$ under different choices of the activation function $\varphi$. Specifically, the role of $\varphi$ in ensuring the algebra $\mathcal{V}$ not only separates points but also universally approximates any function in $C\left(I_n\right)$ could be crucial. Have you explored how the specific properties of $\varphi$, such as smoothness or boundedness, impact the density of $\mathcal{V}$ within $C\left(I_n\right)$? This angle might reveal deeper connections between the algebraic structure of $\mathcal{V}$ and its approximation capabilities.

Answer 2

If I understand correctly, the question is how to prove that the family $$\tag{*} \left\{\sum_{k=1}^M \beta_k \prod_{j=1}^{N_k} \varphi(w_{jk}^T x+\theta_{jk})\ :\ \beta_k, \theta_{jk}\in\mathbb R, w_{jk}\in\mathbb R^n\right\}$$ is an algebra. That is rather obvious for the following reason. Let me rename $$\tag{$\dagger$} f_{w_{jk}, \theta_{jk}}(x):=\varphi(w_{jk}^Tx+\theta_{jk}), $$ and to simplify notation, let me index it by just one parameter: $(f_\alpha)_{\alpha\in A}$. This set $A$ consists of all $w_{jk}$ and all $\theta_{jk}$.

What is the smallest algebra containing all $f_\alpha$? For a set $\mathcal U$ to be an algebra, we need to be able to take linear combinations and pointwise products. So all elements of the form $$\tag{**} \sum_{k=1}^M \beta_k \prod_{j=1}^{N_k} f_{\alpha_{jk}}$$ must be in $\mathcal U$. Conversely, it is clear that the set consisting precisely of these elements is closed under linear combinations and pointwise products. We conclude that $$ \mathcal{U}=\left\{ \sum_{k=1}^M \beta_k \prod_{j=1}^{N_k} f_{\alpha_{jk}}\ :\ M, N_k\in\mathbb N, \beta_k\in\mathbb R\right\}$$ is the smallest algebra containing $(f_\alpha)_{\alpha \in A}$. Which means that our original (*) is the smallest algebra containing the functions of the form $(\dagger)$.

---

I hope this explains where does this set $(*)$ come from. Of course, it remains to prove that $(*)$ contains constants and separates points. The first condition is obvious, and correspond to $N_k=0$. The second depends on some specific property of $\varphi$. For example, the constant function $\varphi(x)=0$ violates it. I suppose that having a nonconstant $\varphi$ is enough to separate points, but that is for another question.