Moore–Aronszajn Theorem
Definition: Let $X$ be a non-empty set. A symmetric function $k:X\times X\to\mathbb{R}$ is called a positive-definite kernel if $$\sum_{i=1}^n\sum_{j=1}^n\xi_i\xi_jk(x_1,x_2)\ge 0$$ for any $n\in\mathbb{N}$ and any $\{x_i\}_{i=1}^n\subset X$ and any $\{\xi_i\}_{i=1}^n\subset \mathbb{R}.$ Here, symmetry means $k(x_1,x_2)=k(x_2,x_1)$ for all $x_1,x_2\in X.$
Definition: Let $X$ be a non-empty set. A Hilbert space $\mathcal{H}$ of real-valued functions on $\mathbb{R}$ is called a Reproducing Kernel Hilbert space (RKHS) if all point-wise evaluation operators are bounded linear operators on $\mathcal{H}.$ Using the Reisz Representation theorem, this is equivalent to the existence of a symmetric positive-definite kernel $k:X\times X\to\mathbb{R},$ called a reproducing kernel, such that
$k_x:X\to\mathbb{R}$ defined as $k_x:y\mapsto k(x,y)$ is a member of $\mathcal{H}$ for all $x\in X$
$f(x)=\langle f,k_x\rangle_\mathcal{H}$ for all $f\in\mathcal{H}$ and $x\in X$
Moore–Aronszajn Theorem: Given any non-empty set $X$ and symmetric positive-definite kernel $k:X\times X\to\mathbb{R},$ there exists a Hilbert space $\mathcal{H}$ of real-valued functions on $X$ with $k$ as its reproducing kernel.
Blueprint of Proof: In a previous post, I have written about the standard completion of general metric spaces. Here, I will use that theorem, to construct the said Hilbert space. The general route is as follows:
Find a pre-Hilbert space of functions $\mathcal{H}_0$.
Find the standard completion of metric space $\mathcal{H}_0,$ which we will call $\mathcal{H}_c$. This is the space of equivalence classes of Cauchy sequences in $\mathcal{H}_0$ under a particular equivalence relation.
Map $\mathcal{H}_c$, via an injection, to another space of real-valued functions $\mathcal{H}$ on $X$.
Show how the notions of inner-product remain “intact” in $\mathcal{H}$
Proof: We are given that $X$ is a non-empty set and $k:X\times X\to\mathbb{R}$ is a symmetric positive-definite kernel on $X.$ We define $$\mathcal{H}_0=\text{span}\big(\{k_x:x\in X\}\big)\text{ where }k_x:X\to\mathbb{R}\text{ is defined as }k_x:y\mapsto k(x,y)$$
Here, span refers to finite real linear combinations of the said functions. We can define an inner product on $\mathcal{H}_0$ as $$\left\langle\sum_{i=1}^{n}a_ik_{x_i},\sum_{j=1}^{m}b_jk_{y_j}\right\rangle=\sum_{i=1}^{n}\sum_{j=1}^{m}a_ib_jk(x_i,y_j)$$
This is obviously well-defined. If $f=\sum_{i=1}^{n}a_ik_{x_i}=\sum_{i=1}^{n’}a’_ik_{x’_i}$ and $g=\sum_{j=1}^{m}b_jk_{y_j}=\sum_{j=1}^{m’}b’_jk_{y’_j}$ then one notes that $$\sum_{i=1}^{n}\sum_{j=1}^{m}a_ib_jk(x_i,y_j)=\sum_{i=1}^{n}a_i\sum_{j=1}^{m}b_jk(x_i,y_j)=\sum_{i=1}^{n}a_ig(x_i)$$$$=\sum_{i=1}^{n}a_i\sum_{j=1}^{m’}b’_jk(x_i,y’_j)=\sum_{j=1}^{m’}b’_j\sum_{i=1}^{n}a_ik(x_i,y’_j)=\sum_{j=1}^{m’}b’_jf(y’_j)$$$$=\sum_{j=1}^{m’}b’_j\sum_{i=1}^{n’}a’_ik(x’_i,y’_j)=\sum_{i=1}^{n’}\sum_{j=1}^{m’}a’_ib’_jk(x’_i,y’_j)$$
Note that for any $f=\sum_{i=1}^{n}a_ik_{x_i}$ and any $x\in X,$ we have $$\langle f,k_x\rangle=\sum_{i=1}^{n}a_ik(x_i,x)=f(x)\text{ }(\star)$$
This fact helps us show that we now have a pre-Hilbert space. Indeed, by the positive-definiteness of $k,$ we have for any $f\in\mathcal{H}_0$ as defined above, $\langle f,f\rangle=\sum_{i=1}^{n}\sum_{j=1}^{n}a_ia_jk(x_i,x_j)\ge 0.$
Moreover, if $\langle f,f\rangle=0$ then, by the Cauchy-Schwartz inequality and $(\star)$, $$0\le |f(x)|=\left|\langle f,k_x\rangle\right|\le \sqrt{\langle f,f\rangle \langle k_x,k_x\rangle}=0$$ for all $x\in X.$ This implies that $f=0$. The remaining properties of inner-product spaces (pre-Hilbert spaces) are easily verified.
Next, let $(\mathcal{H}_c,d_c)$ be the standard completion of the metric space $\mathcal{H}_0.$ The metric in $\mathcal{H}_0$ is induced by the norm of $\mathcal{H}_0$ which, in turn, is induced by our inner product.
Define $\kappa:\mathcal{H}_c\rightarrow \mathbb{R}^X$ as $$\kappa\big([(f_n)_{n=1}^\infty]\big)=x\mapsto\lim_{n\to\infty}f_n(x)$$
Here, $[(f_n)_{n=1}^\infty]$ denotes the equivalence class of the Cauchy-sequence $(f_n)_{n=1}^\infty\subset \mathcal{H}_0.$ The equivalence relation has been described in a previous article.
We need to check if $\kappa$ is well-defined.
Fix any $x\in X.$ Since $(f_n)_{n=1}^\infty$ is a Cauchy sequence, for any $\epsilon>0,$ there exists an $N\in\mathbb{N}$ such that $||f_n-f_m||<\epsilon/||k_x||$ for all $n,m\ge N$ assuming $||k_x||>0.$ Now, using $(\star),$ for any $x\in X,$ we have $$|f_n(x)-f_m(x)|=\langle f_n-f_m,k_x\rangle\le ||f_n-f_m|| ||k_x||<\epsilon$$ Obviously, if $k_x=0,$ then the inequality holds trivially.
Hence, $\{f_n(x)\}_{n=1}^\infty$ is a Cauchy sequence in $\mathbb{R}$ which is complete. Thus, $\lim_{n\to\infty}f_n(x)$ exists. $(\dagger_1)$
Now, if $[(f_n)_{n=1}^\infty]=[(g_n)_{n=1}^\infty]$ then by definition of our equivalence relation, $d_c\big([(f_n)_{n=1}^\infty],[(g_n)_{n=1}^\infty]\big)=\lim_{n\to\infty}||f_n-g_n||=0.$ Hence, by $(\star)$, we have $0\le |f_n(x)-g_n(x)|=\langle f_n-g_n,k_x\rangle\le ||f_n-g_n|| ||k_x||.$ The squeeze theorem and $(\dagger_1)$ thereby imply that $\lim_{n\to\infty}f_n(x)=\lim_{n\to\infty}g_n(x).$ $(\ddagger_1)$
Since $x\in X$ is arbitrary, $(\dagger_1)$ and $(\ddagger_1)$ show that $\kappa$ is, indeed, well-defined.
At this point, we are ready to construct our final space $\mathcal{H}.$ We define $\mathcal{H}=\kappa(\mathcal{H}_c)$, that is the range of $\kappa.$
We equip $\mathcal{H}$ with an inner-product $$\langle f,g\rangle_{\mathcal{H}}=\lim_{n\to\infty}\langle f_n,g_n\rangle\text{ where }f=\kappa\big([(f_n)_{n=1}^\infty]\big)\text{ and }g=\kappa\big([(g_n)_{n=1}^\infty]\big)$$
We need to check if this is well-defined.
Note that $$\left|\langle f_n,g_n\rangle-\langle f_m,g_m\rangle\right|=\left|\big(\langle f_n-f_m,g_n\rangle+\langle f_m,g_n\rangle\big)+\big(\langle f_m,g_n-g_m\rangle-\langle f_m,g_n\rangle\big)\right|$$$$=\left|\langle f_n-f_m,g_n\rangle+\langle f_m,g_n-g_m\rangle\right|\le ||f_n-f_m||||g_n||+||g_n-g_m||||f_m||$$
Since $(f_n)_{n=1}^\infty$ and $(g_n)_{n=1}^\infty$ are Cauchy, they must be bounded. Let $M>0$ be a common bound. For any $\epsilon>0,$ there exists $N_1,N_2\in\mathbb{N}$ such that $||f_n-f_m||<\epsilon/(2M)$ for all $n,m\ge N_1$ and $||g_n-g_m||<\epsilon/(2M)$ for all $n,m\ge N_2.$ Hence, for all $n,m\ge \max\{N_1,N_2\},$ we have $\left|\langle f_n,g_n\rangle-\langle f_m,g_m\rangle\right|<\epsilon.$ Thus, $\{\langle f_n,g_n\rangle\}_{n=1}^\infty$ is a Cauchy sequence in $\mathbb{R}$ which is complete. Hence, the limit exists. $(\dagger_2)$
Moreover, if $[(f_n)_{n=1}^\infty]=[(f’_n)_{n=1}^\infty]$ and $[(g_n)_{n=1}^\infty]=[(g’_n)_{n=1}^\infty]$ then, $$\left|\langle f_n,g_n\rangle-\langle f’_n,g’_n\rangle\right|=\left|\big(\langle f_n-f’_n,g_n\rangle+\langle f’_n,g_n\rangle\big)+\big(\langle f’_n,g_n-g’_n\rangle-\langle f’_n,g_n\rangle\big)\right|$$
$$=\left|\langle f_n-f’_n,g_n\rangle+\langle f’_n,g_n-g’_n\rangle\right|\le ||f_n-f’_n||||g_n||+||g_n-g’_n||||f’_n||$$
Since all the four sequences involved are Cauchy, they must be bounded. Let $M’>0$ be a common bound. For any $\epsilon>0,$ there exists $N’_1,N’_2\in\mathbb{N}$ such that $||f_n-f’_n||<\epsilon/(2M’)$ for all $n\ge N’_1$ and $||g_n-g’_n||<\epsilon/(2M’)$ for all $n\ge N_2$ by definition of our equivalence relation (i.e. $d_c\big([(f_n)_{n=1}^\infty],[(f’_n)_{n=1}^\infty]\big)=\lim_{n\to\infty}||f_n-f’_n||=0$ and likewise for the other pair of sequences). Hence, for all $n\ge \max\{N’_1,N’_2\},$ we have $\left|\langle f_n,g_n\rangle-\langle f’_n,g’_n\rangle\right|<\epsilon.$ Hence, $(\dagger_2)$ implies that $\lim_{n\to\infty}\langle f_n,g_n\rangle=\lim_{n\to\infty}\langle f’_n,g’_n\rangle.$ $(\ddagger_2)$
Thus, $(\dagger_2)$ and $(\ddagger_2)$ show that the operation is, indeed, well-defined.
For any $x\in X,$ the constant sequence $(k_x,k_x,\dots)$ obviously Cauchy in $\mathcal{H}_0$ and $\kappa\big([(k_x,k_x,\dots)]\big)=k_x.$ So $k_x\in\mathcal{H}.$ $(\star\star)$
Now, for any $f=\kappa\big([(f_n)_{n=1}^\infty]\big),$ it follows from $(\star)$ and the definition of $\kappa$ that $$\langle f,k_x\rangle_{\mathcal{H}}=\lim_{n\to\infty}\langle f_n,k_x\rangle=\lim_{n\to\infty}f_n(x)=f(x)\text{ }(\star\star\star)$$
Now, for any $f$ as defined above, $\langle f,f\rangle_{\mathcal{H}}=\lim_{n\to\infty}\langle f_n,f_n\rangle\ge 0$ since $\langle f_n,f_n\rangle\ge 0$ for all $n\in\mathbb{N}.$
Moreover, if $\langle f,f\rangle_{\mathcal{H}}=0$ then by the Cauchy-Schwartz inequality and $(\star\star\star)$, $$0\le |f(x)|=\left|\langle f,k_x\rangle\right|\le \sqrt{\langle f,f\rangle \langle k_x,k_x\rangle}=0$$ for all $x\in X.$ This implies that $f=0$. The remaining properties of inner-product spaces are easily verified.
Note that the norm in this space looks similar: $||f||_{\mathcal{H}}=\sqrt{\langle f,f\rangle_{\mathcal{H}}}=\lim_{n\to\infty}\sqrt{\langle f_n,f_n\rangle}=\lim_{n\to\infty}||f_n||.$ This implies that $$||f-g||_{\mathcal{H}}=d_c\big([(f_n)_{n=1}^\infty],[(g_n)_{n=1}^\infty]\big)\text{ for }f=\kappa\big([(f_n)_{n=1}^\infty]\big)\text{ and }g=\kappa\big([(g_n)_{n=1}^\infty]\big)$$
Now, we will show is that the inner-product space $\mathcal{H}$ is complete.
Let $(F_j)_{j=1}^\infty$ be a Cauchy sequence in $\mathcal{H}.$ Say, $F_j=\kappa\big([(f^j_n)_{n=1}^\infty]\big)$ for each $j\in\mathbb{N}.$
Now, $d_c\big([(f^j_n)_{n=1}^\infty],[(f^k_n)_{n=1}^\infty]\big)=||F_j-F_k||_{\mathcal{H}}$ so $\big([(f^j_n)_{n=1}^\infty]\big)_{j=1}^\infty$ is Cauchy in $\mathcal{H}_c$ which is complete. This latter sequence must converge to some $[(f_n)_{n=1}^\infty]\in \mathcal{H}_c.$ Now, it easily follows that $(F_j)_{j=1}^\infty$ converges to $\kappa\big([(f_n)_{n=1}^\infty]\big).$
Hence, we have a Hilbert space. This fact, together with $(\star\star)$ and $(\star\star\star),$ proves the theorem.
