Supporting vectors vs. principal components

1.   Introduction

Supporting Vector Analysis (SVA) is a relatively recent technique that allows one to solve analytically many real-life problems that used to be tackled by means of Heuristic methods. The lack of mathematical formalism of Heuristic methods resulted many times in unpredictable solutions, that is, mathematical solutions whose real-life interpretations make no sense. Supporting vectors came into play to overcome this issue. This way, supporting vectors were used in a successful way to solve multiobjective optimization problems coming from different disciplines, such as Bioengineering, Physics, and Statistics ^{[4,6,7,8,9,15,22]}, improving considerably the results achieved by other methods like, for instance, Heuristic techniques ^{[10,11,20,21]}.

In ^[4,6,15], it was proven that Singular Value Decomposition (SVD) can be seen as a particular case of SVA. This fact triggered the new trend of restating Statistical notions from the perspective of Functional Analysis and Operator Theory. The main objective of this manuscript is to study Principal Component Analysis (PCA) by means of SVA.

2.   Materials and methods

We will review several basic notions from Operator Theory that will turn out to be crucial for the development of this manuscript.

2.1.   Centering and standardizing

If $x = (x_1, \dots, x_n)\in \mathbb{R}^n$, then the mean of $x$ is defined as $\overline{x}: = \frac{1}{n}\sum_{i = 1}^nx_i$, and its standard deviation is given by $s_x: = \sqrt{\frac{1}{n}\sum_{i = 1}^n \left(x_i-\overline{x}\right)^2}$. Notice that

where ${\bf \overline{x}}: = \left(\overline{x}, \stackrel{n}{\dots}, \overline{x}\right)$ denotes the constant vector of term $\overline{x}$ (in general, if $a\in \mathbb{R}$, then ${\bf a }: = \left(a, \stackrel{n}{\dots}, a\right)$ denotes the constant vector of term $a$).

We say that $x\in \mathbb{R}^n$ is centered provided that $\overline{x} = 0$, and it is standardized provided that $\overline{x} = 0$ and $s_x = 1$. In the latter situation, $\|x\|_2 = \sqrt{n}$, in view of (2.1.1). The subset of centered vectors of $\mathbb{R}^n$ is usually denoted by $\mathrm{cen}(\mathbb{R}^n)$, that is, $\mathrm{cen}(\mathbb{R}^n): = \left\{x\in \mathbb{R}^n: \overline{x} = 0\right\}.$ The subset of standardized vectors of $\mathbb{R}^n$ is usually denoted by $\mathrm{stan}(\mathbb{R}^n)$, that is,

According to (2.1.1), $\mathrm{stan}\left(\mathbb{R}^n\right)\subseteq \sqrt{n} {\mathsf{S}}_{\ell_2^n}$, where ${\mathsf{S}}_{\ell_2^n}$ stands for the unit sphere of $\ell_2^n: = \left(\mathbb{R}^n, \|\cdot\|_2\right)$. In Topology, ${\mathsf{S}}_{\ell_2^n}$ is denoted as ${\mathsf{S}}^{n-1}$.

2.2.   Principal component analysis

The covariance of two vectors $x, y\in \mathbb{R}^n$ is defined as

Notice that $s_{x, x} = s_x^2$, that is, the variance of $x$. The covariance matrix of a given matrix $A\in\mathcal{M}_{m\times n}$ is defined by $s_{{\bf a_1}, \dots, {\bf a_n}}: = \left(s_{{\bf a_i}, {\bf a_j}}\right)_{i, j = 1, \dots, n}$, where ${\bf a_1}, \dots, {\bf a_n}$ stand for the column vectors of $A$.

Consider a matrix $A\in\mathcal{M}_{m\times n}$. The principal components of $A$ are defined as $Ax_1, \dots, Ax_n$, where $\{x_1, \dots, x_n\}$ is an ordered orthonormal basis of eigenvectors of $s_{{\bf a_1}, \dots, {\bf a_n}}$, sorting the eigenvalues of $s_{{\bf a_1}, \dots, {\bf a_n}}$ decreasingly.

We refer the reader to ^[24] for a wider perspective on PCA. Interesting applications of PCA to certain Engineering fields, such as video processing and Big Data, have been provided in ^[3,12].

2.3.   Supporting vector analysis

Let $X, Y$ be Banach spaces. Let $T:X\to Y$ be a bounded linear operator. The operator norm of $T$ is given by

The vector space $\mathscr{C} \mathcal{L}(X, Y)$ of continuous linear operators from $X$ to $Y$ becomes a Banach space when endowed with the operator norm (2.3.1). In the case $X = Y$, $\mathscr{C} \mathcal{L}(X, Y)$ is simply denoted as $\mathscr{C} \mathcal{L}(X)$. If $Y = \mathbb{K}$ ($\mathbb{R}$ or $\mathbb{C}$), then $\mathscr{C} \mathcal{L}(X, Y)$ is denoted as $X^*$, that is, the dual space of $X$. It is also common to denote $\mathscr{C} \mathcal{L}(X, Y)$ by $\mathcal{B}(X, Y)$ and $\mathscr{C} \mathcal{L}(X)$ by $\mathcal{B}(X)$.

The supporting vector notion was formally posed for the first time in ^[5]. However, this concept can be found implicitly and scattered throughout the literature of Banach Space Theory ^[1,2,18,19].

The set of supporting vectors of a bounded linear operator $T:X\to Y$ between Banach spaces $X, Y$ is defined by

Here, ${\mathsf{S}}_X$ stands for the unit sphere of $X$, and ${\mathsf{B}}_X$ denotes the (closed) unit ball of $X$. In the infinite-dimensional setting, it may occur that (2.3.2) is empty. Note that $\mathrm{suppv}(T) = \mathrm{suppv}(\lambda T)$ for all $\lambda\in \mathbb{K}\setminus\{0\}$, and $\mathrm{suppv}(T) = {\mathsf{S}}_{ \mathbb{K}} \mathrm{suppv}(T)$, where $\mathbb{K} = \mathbb{R}\ {\rm{ or }}\ \mathbb{C}$. For a topological and geometrical analysis of the above set, we strongly refer the reader to ^[13,14,23].

For linear functionals, a special subset of supporting vectors is worth regarding. Consider a continuous linear functional $f\in X^*$ in the dual $X^*$ of a Banach space $X$. We define the set of $1$-supporting vectors of $f$ by

Notice that $1$-supporting vectors are particular cases of supporting vectors; in other words, $\mathrm{suppv}_{\mathtt{1}}(f)\subseteq \mathrm{suppv}(f)$. In the upcoming sections, $1$-supporting vectors will be very much relied on. The following remark highlights a standard geometrical property satisfied by $1$-supporting vectors.

Remark 2.1. Consider a Banach space $X$ and a nonzero linear functional $f\in X^*\setminus\{0\}$. For every $x, y\in \mathrm{suppv}_{\mathtt{1}}(f)$ and every $\lambda \in[0, 1]$, we have that $\lambda x+(1-\lambda)y\in \mathrm{suppv}_{\mathtt{1}}(f)$, that is, $\mathrm{suppv}_{\mathtt{1}}(f)$ is a convex subset of the unit sphere ${\mathsf{S}}_X$ of $X$.

A direct consequence of Remark 2.1 is that $\mathrm{suppv}_{\mathtt{1}}(f)$ is either empty or a singleton in strictly convex Banach spaces, like, for instance, Hilbert spaces.

2.4.   Hilbert space theory

Representation Theory is one of the most important theories in Mathematics. A major result in Representation Theory is undoubtedly the Riesz Representation Theorem. This is a key result in Functional Analysis and is crucial for working with self-adjoint operators on Hilbert spaces.

Riesz Representation Theorem. In a Hilbert space $H$, for every $h^*\in H^*$ there exists a unique $h\in H$ satisfying $h^* = (\cdot \lvert h)$. This assignment between $H$ and $H^*$ is a surjective linear isometry.

In view of Remark 2.1 and under the settings of the Riesz Representation Theorem, for every $h\in H\setminus\{0\}$, we have that $\mathrm{suppv}_{\mathtt{1}}\left(h^*\right) = \left\{\frac{h}{\|h\|}\right\}$, that is, $\frac{h}{\|h\|}$ is the only $1$-supporting vector of $h^*$.

On the other hand, the orthogonal subspace of a closed subspace $V$ of a Hilbert space $H$ is denoted by $V^{\perp}$. The orthogonal projection of $H$ onto $V$ is usually denoted by $p_V$. Observe that $H = V\oplus_2 V^{\perp}$, that is, for each $h\in H$,

The adjoint of a bounded linear operator $T:H\to K$ between Hilbert spaces $H, K$ is defined as the unique bounded linear operator $T^*:K\to H$ satisfying $(T(h)\lvert k) = \left(h\lvert T^*(k)\right)$ for each $h\in H$ and each $k\in K$. Basic properties satisfied by the adjoint operator are $\|T^*\| = \|T\|$, $\left(T^*\right)^* = T$, $(T+S)^* = T^*+S^*$, $(T\circ S)^* = S^*\circ T^*$ and $(\lambda T)^* = \overline{\lambda }T^*$.

Lemma 2.2. Every bounded linear operator $T:H\to K$ between Hilbert spaces $H, K$ verifies that $\mathrm{cl}(T(H)) = \ker\left(T^*\right)^{\perp}$ and $T(H)^{\perp} = \mathrm{cl}(T(H))^{\perp} = \ker\left(T^*\right)$.

Proof. First off, it is a trivial observation that $T(H)^{\perp} = \mathrm{cl}(T(H))^{\perp}$. Fix an arbitrary $h\in H$. For every $k\in \ker\left(T^*\right)$, $(T(h)\lvert k) = \left(h\lvert T^*(k)\right) = (h\lvert 0) = 0$. This shows that $T(h)\in\ker\left(T^*\right)^{\perp}$. The arbitrariness of $h\in H$ means that $T(H)\subseteq \ker\left(T^*\right)^{\perp}$. By taking orthogonal complements, we obtain that $\ker\left(T^*\right)\subseteq T(H)^{\perp}$. Conversely, fix an arbitrary $k\in T(H)^{\perp}$. For every $h\in H$, $0 = (T(h)\lvert k) = \left(h\lvert T^*(k)\right)$. This shows that $T^*(k) = 0$, and hence $k\in\ker\left(T^*\right)$. The arbitrariness of $k\in T(H)^{\perp}$ means that $T(H)^{\perp}\subseteq \ker\left(T^*\right)$. By taking orthogonal complements, we finally obtain that $\ker\left(T^*\right)^{\perp}\subseteq \mathrm{cl}(T(H))$.

A bounded operator $T\in \mathcal{B}(H)$ is said to be self-adjoint provided that $T^* = T$. If $H$ is complex, then $T\in \mathcal{B}(H)$ is self-adjoint if and only if $(T(h)|h)\in \mathbb{R}$ for all $h\in H$. A self-adjoint operator is called positive provided that $(T(h)|h)\geq 0$ for all $h\in H$.

For every $T\in \mathcal{B}(H)$, $\sigma(T): = \left\{\lambda\in \mathbb{C}: \lambda I- T \notin \mathcal{U}\left(\mathcal{B}(H)\right)\right\}$ is the spectrum of $T$, where $\mathcal{U}\left(\mathcal{B}(H)\right)$ is the multiplicative group of invertible operators on $H$. Among other spectral properties, the spectrum is compact and nonempty, and $\|T\|\geq \max|\sigma(T)|$. A special subset of the spectrum, called the point spectrum, $\sigma_p(T): = \left\{\lambda\in \mathbb{C}: \ker(\lambda I- T) \neq \{0\}\right\},$ whose elements are called the eigenvalues of $T$, will be very much employed. Note that $\sigma_p(T)\subseteq \sigma(T)$. Furthermore, for each $\lambda \in \sigma_p(T)$, $V_T(\lambda): = \left\{h\in H: T(h) = \lambda h\right\}$ stands for the subspace of eigenvectors associated with $\lambda$. In case there is no confusion with $T$, we will simply denote $V_T(\lambda)$ by $V(\lambda)$.

Suppose next that $\|T\|$ is an eigenvalue of $T$, that is, $\|T\|\in \sigma_p(T)$. In this situation, since $\|T\|\geq \max|\sigma(T)|$, we conclude that $\|T\|$ is the maximum of $|\sigma(T)|$; in other words, $\|T\| = \max|\sigma(T)|$. In this case, we write $\|T\| = \lambda_{\max}(T)$. Observe also that $V(\|T\|)\cap {\mathsf{S}}_X\subseteq \mathrm{suppv}(T)$. Indeed, if $x\in V(\|T\|)\cap {\mathsf{S}}_X$, then $T(x) = \|T\|x$, so $\|T(x)\| = \|T\|$, and hence $x\in \mathrm{suppv}(T)$.

Nevertheless, in general, $\|T\|\notin \sigma_p(T)$, unless, for instance, $T$ is compact, self-adjoint and positive. This is why we have to rely on the adjoint $T^*$ and on the strongly positive operator $T^*\circ T$. It is straightforward to check that the eigenvalues of a self-adjoint operator are real, and the eigenvalues of a self-adjoint positive operator are positive. When $T$ is compact, it holds that $T^*\circ T$ is compact, self-adjoint and positive.

The following result, on which we will strongly rely later on, can be found in [6,Theorem 4], which is itself a refinement of [14,Theorem 9].

Theorem 2.3. Let $H, K$ be Hilbert spaces. Let $T\in \mathcal{B}(H, K)$. Then, $\|T\|^2 = \|T^*\circ T\|$, and $\mathrm{suppv}(T)\subseteq \mathrm{suppv}\left(T^*\circ T\right)$. Furthermore, $\mathrm{suppv}(T)\neq \varnothing$ if and only if $\|T^*\circ T\|\in \sigma_p\left(T^*\circ T\right)$. In this situation, $\|T\| = \sqrt{\lambda_{\max}\left(T^*\circ T\right)}$, and $\mathrm{suppv}(T) = V\left(\lambda_{\max}(T^*\circ T)\right)\cap {\mathsf{S}}_H$.

3.   Results

In this section, we will state and prove all the novel theorems of this work. This section is divided into four subsections, in which we will deal with supporting vectors, principal components and the topological structure of the subsets of centered and standardized vectors.

3.1.   Topological structure of the subsets of centered and standardized vectors

This subsection begins unveiling the topological structure of the subsets $\mathrm{cen}(\mathbb{R}^n)$ and $\mathrm{stan}(\mathbb{R}^n)$ of centered and standardized vectors, respectively. Notice that $\mathrm{cen}(\mathbb{R}) = \{0\}$ and $\mathrm{stan}(\mathbb{R}) = \varnothing$.

Theorem 3.1. If $n\geq 1$, then $\mathrm{cen}(\mathbb{R}^n)$ is linearly isometric, and hence homeomorphic, to $\ell_2^{n-1}$. If $n\geq 2$, then $\mathrm{stan}(\mathbb{R}^n)$ is linearly isometric, and hence homeomorphic, to ${\mathsf{S}}^{n-2}$.

Proof. Notice that $\overline{x+y} = \overline{x}+\overline{y}$, and $\overline{tx} = t\overline{x}$ for all $x\in \mathbb{R}^n$ and all $t\in \mathbb{R}$. As a consequence,

is a linear functional (usually called the mean functional). Next, simply observe that $\mathrm{cen}(\mathbb{R}^n) = \ker\left(\overline{\cdot}\right)$. Thus, by bearing in mind (2.1.1), we immediately obtain that $\mathrm{stan}(\mathbb{R}^n) = \ker\left(\overline{\cdot}\right)\cap \sqrt{n} {\mathsf{S}}_{\ell_2^n} = \sqrt{n} {\mathsf{S}}_{\ker\left(\overline{\cdot}\right)}$. In other words, $\mathrm{stan}(\mathbb{R}^n)$ is precisely a multiple of the unit sphere of the Hilbert subspace $\ker\left(\overline{\cdot}\right)$ of $\ell_2^n$. Since $\dim\left(\ker\left(\overline{\cdot}\right)\right) = n-1$, we have that $\ker\left(\overline{\cdot}\right)$ is linearly isometric to $\ell_2^{n-1}$. As a consequence, the unit sphere of $\ker\left(\overline{\cdot}\right)$, $\frac{1}{\sqrt{n}} \mathrm{stan}(\mathbb{R} ^n)$, is linearly isometric to the unit sphere of $\ell_2^{n-1}$, ${\mathsf{S}}_{\ell_2^{n-1}} = {\mathsf{S}}^{n-2}$.

The following lemma aims at computing the norm of the mean functional as an element of the dual space of $\ell_2^n$ as well as its only $1$-supporting vector.

Lemma 3.2. In $\left(\ell_2^n\right)^*$, $\left\|\overline{\cdot}\right\| = \frac{1}{\sqrt{n}}$ and $\mathrm{suppv}_{\mathtt{1}}\left(\overline{\cdot}\right) = \left\{{ \frac{\bf{1}}{\sqrt{n}}}\right\}$.

Proof. Hölder's Inequality ensures that

for every $x\in\ell_2^n$. This shows that $\left\|\overline{\cdot}\right\|\leq \frac{1}{\sqrt{n}}$. On the other hand, $\|{\bf 1} \|_2 = \sqrt{n}$, that is, ${ \frac{\bf{1}}{\sqrt{n}}}\in {\mathsf{S}}_{\ell_2^n}$. Finally,

As a consequence, $\left\|\overline{\cdot}\right\| = \frac{1}{\sqrt{n}}$ and $\mathrm{suppv}_{\mathtt{1}}\left(\overline{\cdot}\right) = \left\{{ \frac{{\bf{1}}}{\sqrt{n}}}\right\}$.

As a direct consequence of Lemma 3.2, the standard deviation of a vector $x\in \mathbb{R}^n$ can be rewritten as the following (2.1.1):

By bearing in mind the Riesz Representation Theorem, Lemma 3.2 assures that if $h: = {\bf \frac{1}{n}}$, then $h^* = \overline{\cdot}$ in $H: = \ell_2^n$. Notice that $h: = {\bf \frac{1}{n}} = \left(\frac{1}{n}, \stackrel{n}{\dots}, \frac{1}{n}\right)$ can be seen as a (finite) convex series. This fact will help us generalize these concepts to an infinite-dimensional separable Hilbert space setting later on in the Discussion.

Definition 3.3 For $n\geq 1$, the mean operator is defined by

and the centering operator is defined as

For $n\geq 2$, the standardizing operator is defined as

It is a trivial observation that

for all $x\in \mathbb{R}^n$.

Recall that a projection on a Banach space $X$ is a continuous, linear, and idempotent map $P:X\to X$. Its dual operator $P^*:X^*\to X^*$ is also a projection. The complementary projection of $P$ is defined as $I_X-P$, which is also a projection. Every non-zero projection has norm greater than or equal to $1$. A $1$-projection is a projection of norm $1$ (also called a contractive projection), and a $(1, 1)$-projection is a $1$-projection whose complementary projection is also a $1$-projection (also called bicontractive). Orthogonal projections in Hilbert spaces are the most representative examples of bicontractive projections.

The final result of this first subsection serves to show that both the mean operator and the centering operator are complementary projections to each other of norm $1$, that is, bicontractive, for the Euclidean norm. Even more, the mean operator and the centering operator are orthogonal projections to each other. This will allow us to directly obtain the König-Huygens Theorem (3.1.7) as a direct consequence of the Pythagorean Theorem in Hilbert spaces. The König-Huygens Theorem provides the classical decomposition of the $2$-norm of a vector of $\mathbb{R}^n$ in terms of the mean and the standard deviation.

Theorem 3.4. The centering operator is a linear projection on $\mathbb{R}^n$ whose kernel is $\ker(\mathrm{cen}) = \mathbb{R}^n{\bf 1}$, whose range is $\mathrm{cen}(\mathbb{R}^n) = \ker\left(\overline{\cdot}\right)$, and whose complementary projection is the mean operator. Furthermore, if we consider $\mathbb{R}^n$ endowed with the Euclidean norm, then $\|{\bf \overline{\bullet}}\| = \| \mathrm{cen}\| = 1$, and ${\bf \overline{\bullet}}$ and $\mathrm{cen}$ are complementary orthogonal projections. As a consequence, for every $x\in \mathbb{R}^n$,

Proof. In the first place, notice that ${\bf \overline{\bullet}}$ is clearly linear, since ${\bf \overline{\bullet}} = \overline{\cdot}{\bf 1}$; in other words, ${\bf \overline{x}} = \left(\overline{x}, \stackrel{n}{\dots}, \overline{x}\right) = \overline{x}\left(1, \stackrel{n}{\dots}, 1\right) = \overline{x}{\bf 1}$ for all $x\in \mathbb{R}^n$. As a consequence, $\mathrm{cen}$ is linear as well because $\mathrm{cen} = I_{ \mathbb{R}^n}-{\bf \overline{\bullet}}$, that is, the difference of two linear operators on $\mathbb{R}^n$. On the other hand, observe that if $x\in \mathbb{R}^n$ is a constant vector, that is, $x = t {\bf 1} = {\bf t}$ for some $t\in \mathbb{R}$, then $\mathrm{cen}(x) = 0$. Conversely, if $\mathrm{cen}(x) = 0$, then $x = {\bf \overline{x}} = \overline{x}{\bf 1}$, and hence $x\in \mathbb{R}^n{\bf 1}$ is a constant vector. This shows that $\ker(\mathrm{cen}) = \mathbb{R}^n{\bf 1}$. From Theorem 3.1, we already know that $\mathrm{cen}(\mathbb{R}^n) = \ker\left(\overline{\cdot}\right)$. Next, let us prove that $\mathrm{cen}$ is a projection. Fix an arbitrary $x\in \mathbb{R}^n$. Notice that

Since ${\bf \overline{\bullet}} = I_{ \mathbb{R}^n}- \mathrm{cen}$, we conclude that the mean operator is the complementary projection to the centering operator. Finally, let us compute $\|{\bf \overline{\bullet}}\|$ and $\| \mathrm{cen}\|$. In accordance with Lemma 3.2, for every $x\in \mathbb{R}^n$ we have that

meaning that $\|{\bf \overline{\bullet}}\|\leq 1$. Next,

This shows that $\|{\bf \overline{\bullet}}\| = 1$. In order to prove that ${\bf \overline{\bullet}}$ and $\mathrm{cen}$ are orthogonal projections, it only suffices to realize that $\mathbb{R}{\bf 1}$ and $\mathrm{cen}(\mathbb{R}^n)$ are orthogonal subspaces. Indeed, for every $t\in \mathbb{R}$ and every $x\in \mathbb{R}^n$ with $\overline{x} = 0$, we have that $(t{\bf 1}| x) = t({\bf 1}| x) = t\sum_{i = 1}^nx_i = tn\overline{x} = 0$, and hence $\mathbb{R}{\bf 1}\subseteq \mathrm{cen}(\mathbb{R}^n)^{\perp}$, or equivalently, $\mathrm{cen}(\mathbb{R}^n)\subseteq \left(\mathbb{R}{\bf 1}\right)^{\perp}$. Furthermore, the Pythagorean Theorem in $\ell_2^n$ allows that

for every $t\in \mathbb{R}$ and every $x\in \mathbb{R}^n$ with $\overline{x} = 0$. Next, if $y\in\left(\mathbb{R}{\bf 1}\right)^{\perp}$, then $0 = \frac{1}{n}({\bf 1}| y) = \frac{1}{n}\sum_{i = 1}^ny_i = \overline{y}$, meaning that $y\in \ker\left(\overline{\cdot}\right) = \mathrm{cen}(\mathbb{R}^n)$. As a consequence, $\mathrm{cen}(\mathbb{R}^n)^{\perp} = \mathbb{R}{\bf 1}$, resulting in

for every $x\in \mathbb{R}^n$. It only remains to show that $\| \mathrm{cen}\| = 1$, but this is a direct consequence of the fact that $\mathrm{cen}$ is an orthogonal projection.

3.2.   Supporting vectors and first principal component

In this subsection, we will provide sufficient conditions for the supporting vectors to coincide with the first principal component. First, we will need some definitions.

Definition 3.5. A matrix is said to be centered (standardized) provided that all of its column vectors are centered (standardized) vectors.

The following lemma displays a simple characterization of centered matrices.

Lemma 3.6. Let $A\in\mathcal{M}_{m\times n}(\mathbb{R})$. The following conditions are equivalent:

1) $A$ is centered.

2) $Ax$ is centered for all $x\in \mathbb{R}^n$.

Proof. Let $\{e_1, \dots, e_n\}$ denote the canonical basis of $\mathbb{R}^n$. Notice that the columns of $A$ are precisely $Ae_j$ for $j = 1, \dots, n$. Suppose first that $A$ is centered. Fix an arbitrary $x\in \mathbb{R}^n$. The linearity of the mean functional allows that

As a consequence, $Ax$ is centered for all $x\in \mathbb{R}^n$. Conversely, suppose now that $Ax$ is centered for all $x\in \mathbb{R}^n$. In particular, $\overline{Ae_j} = 0$ for $j = 1, \dots, n$, meaning that the columns of $A$ are centered vectors, that is, $A$ is centered.

The following characterization of centered matrices is a bit more sophisticated. First, a technical lemma is needed.

Lemma 3.7. Consider $x, y\in \mathbb{R}^m$. Then,

1) $ms_{x, y} = x\bullet y$ if and only if either $x$ or $y$ is centered.

2) $x$ is standardized if and only if $x$ is centered and $x\bullet x = m$.

Proof.

$1)$ Let us observe that

As a consequence, $ms_{x, y} = x\bullet y$ if and only if either $x$ or $y$ is centered.

$2)$ By definition, $x$ is standardized if and only if $x$ is centered and $s_x = 1$. We know that then $\|x-\overline{x}\|_2 = \sqrt{m}s_x$. In the context of centered vectors, the previous expression becomes $\sqrt{x\bullet x} = \|x\|_2 = \sqrt{m}s_x$. Therefore, $x$ is standardized if and only if $x$ is centered and $x\bullet x = m$.

Recall that the covariance matrix of a given matrix $A\in\mathcal{M}_{m\times n}$ is defined by $s_{{\bf a_1}, \dots, {\bf a_n}}: = \left(s_{{\bf a_i}, {\bf a_j}}\right)_{i, j = 1, \dots, n}$, where ${\bf a_1}, \dots, {\bf a_n}$ stand for the column vectors of $A$. Also, recall that if $B$ is a square matrix, then $\mathrm{diag}(B)$ stands for the diagonal of $B$.

Proposition 3.8. Let $A\in\mathcal{M}_{m\times n}(\mathbb{R})$. The following conditions are equivalent:

1) $A$ is centered.

2) $A^tA = ms_{{\bf a_1}, \dots, {\bf a_n}}$.

3) $\mathrm{diag}\left(A^tA\right) = \mathrm{diag}\left(ms_{{\bf a_1}, \dots, {\bf a_n}}\right)$.

Proof. Notice that $A^tA = \left({\bf a_i}\bullet {\bf a_j}\right)_{i, j = 1, \dots, n}$. Suppose first that $A$ is centered. In view of Lemma 3.7, $ms_{{\bf a_i}, {\bf a_j}} = {\bf a_i}\bullet {\bf a_j}$ for all $i, j\in\{1, \dots, n\}$, meaning that $A^tA = ms_{{\bf a_1}, \dots, {\bf a_n}}$. Next, if $A^tA = ms_{{\bf a_1}, \dots, {\bf a_n}}$, then we trivially have that $\mathrm{diag}\left(A^tA\right) = \mathrm{diag}\left(ms_{{\bf a_1}, \dots, {\bf a_n}}\right)$. Finally, assume that $\mathrm{diag}\left(A^tA\right) = \mathrm{diag}\left(ms_{{\bf a_1}, \dots, {\bf a_n}}\right)$. Then, $ms_{{\bf a_i}, {\bf a_i}} = {\bf a_i}\bullet {\bf a_i}$ for all $i = 1, \dots, n$. In accordance with Lemma 3.7, ${\bf a_i}$ is centered for all $i = 1, \dots, n$, meaning that $A$ is centered.

As an immediate consequence of Lemma 3.7 and Proposition 3.8, we obtain the following corollary.

Corollary 3.9. Let $A\in\mathcal{M}_{m\times n}(\mathbb{R})$. The following conditions are equivalent:

1) $A$ is standardized.

2) $A^tA = ms_{{\bf a_1}, \dots, {\bf a_n}}$ and $\mathrm{diag}\left(A^tA\right) = \left(m, \stackrel{n}{\dots}, m\right)$.

Finally, we have gathered all the necessary tools to prove our main results of this subsection. Simply keep in mind the small observation that if $B\in\mathcal{M}_{n}(\mathbb{R})$, then $\sigma_p(\alpha B) = \alpha \sigma_p(B)$ and $V_{\alpha B}(\alpha \lambda) = V_B(\lambda)$ for all $\alpha\in \mathbb{R}\setminus\{ 0\}$ and all $\lambda\in \sigma_p(B)$.

Theorem 3.10. Let $A\in\mathcal{M}_{m\times n}(\mathbb{R})$. If $A$ is centered, then

where $A$ is seen as a linear operator

Proof. According to Theorem 2.3, $\mathrm{suppv}(A) = V\left(\lambda_{\max}(A^*\circ A)\right)\cap {\mathsf{S}}_{\ell_2^n}$. Notice that the adjoint of $A$, $A^*$, coincides with its transpose, $A^t$. On the other hand, since $A$ is centered, Proposition 3.8 allows that $A^tA = ms_{{\bf a_1}, \dots, {\bf a_n}}$. Finally,

We will conclude this subsection with an example of a centered matrix $A$ whose last principal component has at least two dimensions.

Remark 3.11. Let $T:H\to K$ be a bounded linear operator between Hilbert spaces $H, K$. Then, $\ker(T)\subseteq \ker\left(T^*\circ T\right)$. As a consequence, if $0\in\sigma_p(T)$, then $0\in \sigma_p\left(T^*\circ T\right)$.

Notice that if $T\in \mathcal{B}(H)$ is a self-adjoint positive operator on a Hilbert space $H$ such that $\ker(T)\neq \{0\}$, then $0$ is the minimum of $\sigma_p(T)$ since all the eigenvalues of $T$ are positive.

Example 3.12. Let $A\in\mathcal{M}_{m\times n}(\mathbb{R})$ be any centered matrix with three equal columns. Then, $\ker(A)\subseteq \ker(A^tA) = \ker(s_{{\bf a_1}, \dots, {\bf a_n}})$ since $A^tA = ms_{{\bf a_1}, \dots, {\bf a_n}}$ in accordance with Proposition 3.8. Notice that $\ker(A)$ has at least dimension $2$. Observe that $0\in \sigma_p\left(s_{{\bf a_1}, \dots, {\bf a_n}}\right)$. As a consequence, the last principal component of $A$ corresponds to $Ax_1, \dots, Ax_p$, where $\{x_1, \dots, x_p\}$ is an orthonormal basis of $V_{s_{{\bf a_1}, \dots, {\bf a_n}}}(0)$.

3.3.   Second principal component as supporting vector

This subsection is aimed at showing the second principal component of a centered matrix can be obtained via a supporting vector of a derived matrix from the original one.

Theorem 3.13. Let $H, K$ be Hilbert spaces and $T:H\to K$ be a continuous linear operator. Let $H_1$ be a closed subspace of $H$. Then,

1) $\left\|T|_{H_1}\right\| = \left\|T\circ p_{H_1}\right\|$.

2) $\mathrm{suppv}\left(T|_{H_1}\right) = \mathrm{suppv}\left(T\circ p_{H_1}\right)$.

3) $\left(T|_{H_1}\right)^* = p_{H_1}\circ T^*$.

4) If $H_1$ is an invariant subspace of $T^*\circ T$, then $\left(T|_{H_1}\right)^*\circ T|_{H_1} = \left(T^*\circ T\right)|_{H_1}$.

Proof.

$1)$ In the first place, $p_{H_1}$ is an orthogonal projection. Thus, $\left\|p_{H_1}\right\| = 1$, and therefore $\left\|T\circ p_{H_1}\right\| = \left\|T|_{H_1}\circ p_{H_1}\right\|\leq \left\|T|_{H_1}\right\|\left\|p_{H_1}\right\| = \left\|T|_{H_1}\right\|$. On the other hand, for every $h_1\in {\mathsf{B}}_{H_1}$, $\left\|T|_{H_1}(h_1)\right\| = \|T(h_1)\| = \left\|T\left(p_{H_1}\right)\right\| = \left\|\left(T\circ p_{H_1}\right)(h_1)\right\|\leq \left\|T\circ p_{H_1}\right\|$. As a consequence, $\left\|T|_{H_1}\right\|\leq \left\|T\circ p_{H_1}\right\|$, meaning that $\left\|T|_{H_1}\right\| = \left\|T\circ p_{H_1}\right\|$.

$2)$ Fix an arbitrary $h_1\in \mathrm{suppv}\left(T|_{H_1}\right)$. Then, $\left\|\left(T\circ p_{H_1}\right)(h_1)\right\| = \left\|T|_{H_1}(h_1)\right\| = \left\|T|_{H_1}\right\| = \left\|T\circ p_{H_1}\right\|$, meaning that $h_1\in \mathrm{suppv}\left(T\circ p_{H_1}\right)$. Conversely, take $h\in \mathrm{suppv}\left(T\circ p_{H_1}\right)$. We will show first that $p_{H_1}(h)\in \mathrm{suppv}\left(T|_{H_1}\right)$. Indeed, $\left\|T|_{H_1}\left(p_{H_1}(h)\right)\right\| = \left\|\left(T\circ p_{H_1}\right)(h)\right\| = \left\|T\circ p_{H_1}\right\| = \left\|T|_{H_1}\right\|$. Since $\left\|p_{H_1}(h)\right\|\leq \|h\| = 1$, we conclude that $\left\|p_{H_1}(h)\right\| = 1$, and hence $p_{H_1}(h)\in \mathrm{suppv}\left(T|_{H_1}\right)$. Now, notice that

meaning that $p_{H_1^{\perp}}(h) = 0$ and $h = p_{H_1}(h)\in \mathrm{suppv}\left(T|_{H_1}\right)$.

$3)$ Fix arbitrary elements $h_1\in H_1$ and $k\in K$. Notice that

By the uniqueness of the adjoint, we conclude that $\left(T|_{H_1}\right)^* = p_{H_1}\circ T^*$.

$4)$ Finally, let us show that $\left(T|_{H_1}\right)^*\circ T|_{H_1} = \left(T^*\circ T\right)|_{H_1}$ whenever $H_1$ is an invariant subspace of $T^*\circ T$. Indeed, fix an arbitrary $h_1\in H_1$. Then,

If $X$ is a normed space, $T:X\to X$ is a continuous linear operator, and $\lambda\in \sigma_p(T)$, then it is clear that $T\left({\mathsf{V}}(\lambda)\right)\subseteq {\mathsf{V}}(\lambda)$. The following lemma is also well known, yet we include the proof for the sake of completeness.

Lemma 3.14. Let $H$ be a Hilbert space, and $T:H\to H$ is a self-adjoint continuous linear operator. Let $\lambda\in\sigma_p(T)$. Then,

1) $T\left({\mathsf{V}}(\lambda)^{\perp}\right)\subseteq {\mathsf{V}}(\lambda)^{\perp}$.

2) $\sigma_p\left(T|_{{\mathsf{V}}(\lambda)^{\perp}}\right) = \sigma_p(T)\setminus\{\lambda\}$.

Proof.

$1)$ Fix arbitrary elements $w\in {\mathsf{V}}(\lambda)^{\perp}$ and $v\in {\mathsf{V}}(\lambda)$. Observe that

This shows that $T(w)\in {\mathsf{V}}(\lambda)^{\perp}$. The arbitrariness of $w\in {\mathsf{V}}(\lambda)^{\perp}$ serves to assure that $T\left({\mathsf{V}}(\lambda)^{\perp}\right)\subseteq {\mathsf{V}}(\lambda)^{\perp}$.

$2)$ Take any $\gamma\in \sigma_p\left(T|_{{\mathsf{V}}(\lambda)^{\perp}}\right)$. There exists $w\in {\mathsf{V}}(\lambda)^{\perp}\setminus\{0\}$ such that $T(w) = \gamma w$. It is clear that $\gamma\in \sigma_p(T)$. If $\gamma = \lambda$, then $w\in {\mathsf{V}}(\lambda)$, meaning the contradiction that $w = 0$. Conversely, take any $\gamma\in \sigma_p(T)\setminus\{\lambda\}$. There exists $w\in H\setminus\{0\}$ such that $T(w) = \gamma w$. It suffices to show that $w\in {\mathsf{V}}(\lambda)^{\perp}$. Indeed, since $\lambda\neq \gamma$, either $\lambda$ or $\gamma$ is not $0$. We can assume without any loss of generality that $\gamma\neq 0$. Then, for every $v\in {\mathsf{V}}(\lambda)$, we have that

Since $\lambda\neq \gamma$, the only possibility is that $(w|v) = 0$. As a consequence, $w\in {\mathsf{V}}(\lambda)^{\perp}$.

Theorem 3.15. Let $H, K$ be Hilbert spaces and $T:H\to K$ be a compact linear operator. If $\lambda\in \sigma_p\left(T^*\circ T\right)$ is the second largest eigenvalue of $T^*\circ T$, then $\lambda = \left\|T|_{{\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}}\right\|^2$ and

Proof. In the first place, $T^*\circ T:H\to H$ is self-adjoint, positive, and compact. According to Theorem 2.3, $\|T\|^2 = \|T^*\circ T\|\in\sigma_p\left(T^*\circ T\right)$, and $\|T\|^2$ is the largest eigenvalue of $T^*\circ T$. By applying Lemma 3.14, we have that $\left(T^*\circ T\right)\left({\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}\right)\subseteq {\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}$ and

As a consequence, $\lambda$ is the largest eigenvalue of $\left(T^*\circ T\right)|_{{\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}}$. Next, since ${\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}$ is an invariant subspace of $T^*\circ T$, Theorem 3.13(4) allows that

This means that $\lambda$ is the largest eigenvalue of $\left(T|_{{\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}}\right)^*\circ T |_{{\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}}$. By relying again on Theorem 2.3 and on the fact that $T |_{{\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}}$ is also compact, we obtain that $\lambda = \left\|T|_{{\mathsf{V}}_{T^*\circ T}\left(\|T\|^2\right)^{\perp}}\right\|^2$ and

Finally, Theorem 3.13(2) assures that

3.4.   Supporting vectors of quotient operators

Let $X$ be a Banach space and $M\subseteq X$ be a closed subspace. The quotient space of $X$ by $M$ is defined by ${}^{X}\!\!\diagup\!\!{}_{M}\;: = \{x+M:x\in X\}$ and becomes a Banach space endowed with the distance-to-$M$ norm $\|x+M\|: = d(x, M): = \inf\{\|x-m\|:m\in M\}$. The canonical projection of $X$ onto ${}^{X}\!\!\diagup\!\!{}_{M}\;$ is given by

and it is clearly a linear operator of norm $\|\pi_M\| = 1$. The closed subspace $M$ is called proximinal provided that for every $x\in X$, the distance from $x$ to $M$ is attained at some $m_0\in M$, that is, $d(x, M) = \|x-m_0\|$.

Let $X, Y$ be Banach spaces and $T:X\to Y$ be a continuous linear operator. Let $M\subseteq \ker(T)$ be a closed subspace. The quotient operator of $T$

is a well-defined continuous linear operator. Notice that when $M$ is chosen to be $\ker(T)$, we obtain the First Isomorphism Theorem. The following theorem relates the supporting vectors of $T$ with those of $T_M$.

Theorem 3.16. Let $X, Y$ be Banach spaces and $T:X\to Y$ be a continuous linear operator. Let $M\subseteq \ker(T)$ be a closed subspace. Then,

1) $\left\|T_M\right\| = \|T\|$.

2) $\pi_M\left(\mathrm{suppv}(T)\right)\subseteq \mathrm{suppv}\left(T_M\right)$.

3) If $M$ is proximinal, then $\pi_M\left(\mathrm{suppv}(T)\right) = \mathrm{suppv}\left(T_M\right)$.

Proof.

$1)$ Indeed, if $\|x\|\leq 1$, then $\|x+M\|: = d(x, M)\leq \|x-0\|\leq 1$. Therefore, $\|T(x)\| = \|T_M(x+M)\|\leq \|T_M\|$. This shows that $\|T\|\leq \|T_M\|$. Conversely, if $\|x+M\| = 1$, then we can find a sequence $(m_k)_{k\in \mathbb{N}}\subseteq M$ such that $\|x-m_k\|\to d(x, M) = \|x+M\| = 1$ as $k\to \infty$. Notice that $\left\|T\left(\frac{x-m_k}{\|x-m_k\|}\right)\right\|\leq \|T\|$ for all $k\in \mathbb{N}$, that is, $\|T_M(x+M)\| = \|T(x)\| = \left\|T\left(x-m_k\right)\right\|\leq \|T\|\|x-m_k\|$ for all $k\in \mathbb{N}$. Since $\|x-m_k\|\to 1$ as $k\to \infty$, we conclude that $\|T_M(x+M)\|\leq \|T\|$. The arbitrariness of $x+M\in {\mathsf{S}}_{{}^{X}\!\!\diagup\!\!{}_{M}\;}$ assures that $\left\|T_M\right\|\leq \|T\|$.

$2)$ Fix an arbitrary $x\in \mathrm{suppv}(T)$. Notice that $\|x\| = 1$, so $\|x+M\| = d(x, M)\leq \|x-0\| = \|x\| = 1$. This shows that $x+M\in {\mathsf{B}}_{{}^{X}\!\!\diagup\!\!{}_{M}\;}$. Next, $\|T_M(x+M)\| = \|T(x)\| = \|T\| = \|T_M\|$. As a consequence, $\|x+M\|\in \mathrm{suppv}(T_M)$.

$3)$ Fix an arbitrary $x+M\in \mathrm{suppv}(T_M)$. Since $M$ is proximinal, there exists $m_0\in M$ such that $1 = \|x+M\| = d(x, M) = \|x-m_0\|$. We will show that $x-m_0\in \mathrm{suppv}(T)$. Indeed, $\|x-m_0\| = 1$, and $\|T(x-m_0)\| = \|T(x)\| = \|T_M(x+M)\| = \|T_M\| = \|T\|$. Finally, $\pi_M(x-m_0) = x+M$.

3.5.   Applications of SVA/PCA to real-life situations

In this section, we present the application of our SVA/PCA theorems to solve a real-life problem focused on the distribution of political sensitivities with several economic variables. For that, we have used the results of the 2018 Andalusian (Spain) elections. The data analyzed is provided by the Institute of Statistics and Cartography of Andalusia ^[17] and combines information from 153 municipalities with more than ten thousand inhabitants and 8 measured variables. The data include unemployment rate, aging rate and 6 generalist policy options ranging from left to right wing. Note that the variables in columns were previously centered.

A Graphical User Interface (GUI) has been developed in Python code and aims to compute the eigenvectors, including the supporting vector (first eigenvector), and the principal components. Figure 1 shows the GUI of the PCA input, where the data matrix is introduced. It is important to note that the input data can be imported by a CSV file. This option is very useful when we are working with large matrices. The results obtained are shown in Figure 2, which can also be exported as a CSV file.

First of all, we note that the $A\in\mathcal{M}_{153\times 8}(\mathbb{R})$ data matrix is centered in the origin by subtracting the mean. Then, we calculate the covariance matrix for the centered matrix. Afterwards, we compute the eigenvalues and eigenvectors. In particular, as mentioned in this paper, the eigenvector associated with the maximum eigenvalue corresponds to the supporting vector. These eigenvalues and eigenvectors are sorted in descending order, so the first element corresponds to the supporting vector and the first principal component. Finally, the products of the initial matrix and the eigenvectors are done, obtaining the principal components.

In particular, focusing on the example of the Andalusian elections, the results show the correspondence of the first principal component with the classic political sensitivities associated with old-aging municipalities. On the other hand, the second principal component is related to the differentiation between left-wing and right-wing political options.

In Figure 3, the 8 variables used in the example are shown in the eigenvector reference system with the supporting vector and the second eigenvector. The representation of the 153 municipalities in the principal component reference system ($Ax$) is shown in Figure 4. This 2D image with the first and the second principal component group the municipalities according to the political sensitivities and their demographic characteristics.

4.   Discussion

We will discuss how to transport the mean functional, the mean operator and the centering operator to abstract settings, such as Hilbert spaces, Banach algebras and probability spaces.

4.1.   Generalization to Hilbert spaces

As mentioned in Lemma 3.2, in view of the Riesz Representation Theorem, if $h_0: = {\bf \frac{1}{n}} = \left(\frac{1}{n}, \stackrel{n}{\dots}, \frac{1}{n}\right)$, then $h_0^* = \overline{\cdot}$ is precisely the mean functional in $H: = \ell_2^n$. Actually, $\|h_0\|_2 = \frac{1}{\sqrt{n}} = \|h_0^*\|$. At this stage, the key is to realize that $h_0$ can be seen as a (finite) convex series. Recall that a convex series is a convergent series $\sum_{n = 1}^\infty t_n$ such that $\sum_{n = 1}^\infty t_n = 1$ and $t_n\geq 0$ for all $n\in \mathbb{N}$. Notice that if $\sum_{n = 1}^\infty t_n$ is a convex series, then $\sum_{n = 1}^\infty t_n^2\leq \sum_{n = 1}^\infty t_n = 1$, and hence $(t_n)_{n\in \mathbb{N}}\in\ell_2$. Now, we are in the right position to define the notions of mean and standard deviation on separable Hilbert spaces.

Let $H$ be a separable Hilbert space with an orthonormal basis $(e_n)_{n\in \mathbb{N}}$. Fix a convex series $\sum_{n = 1}^\infty t_n$. Let $h\in H$ and write $h = \sum_{n = 1}^\infty \left(h|e_{n}\right)e_{n}$. The mean of $h$, with respect to $(t_n)_{n\in \mathbb{N}}$, is defined as

The mean functional, with respect to $(t_n)_{n\in \mathbb{N}}$, is given by

By relying on Hölder's Inequality, it is not hard to check that the mean functional is an element of $H^*$ whose norm is precisely $\sqrt{\sum_{n = 1}^\infty t_n^2} = \left\|(t_n)_{n\in \mathbb{N}}\right\|_2$. In accordance with the Riesz Representation Theorem, there exists $h_0\in H$ such that $(h|h_0) = \overline{h}$ for all $h\in H$. If we let $h = \sum_{n = 1}^\infty \left(h|e_{n}\right)e_{n}$, then we obtain that

By taking $h = e_{n}$ for every $n\in \mathbb{N}$ in (4.1.3), we conclude that $(e_n|h_0) = t_n$ for every $n\in \mathbb{N}$. In particular, $h_0 = \sum_{n = 1}^\infty t_ne_n$. As expected,

In view of the Riesz Representation Theorem, $h_0^* = (\cdot|h_0) = \overline{\cdot}$. Notice that, in order to conclude that $h_0^* = (\cdot|h_0) = \overline{\cdot}$, the Riesz Representation Theorem is really not needed. Let us get back for a second to $\ell_2^n$. For every $x\in\ell_2^n$,

Then, going back to a general separable Hilbert space $H$, the mean operator is defined as

It is not hard to check that the mean operator is an orthogonal projection on $H$ whose complementary projection is, precisely, the centering operator:

4.2.   Generalization to Banach algebras

A Banach algebra is a real or complex algebra $A$ endowed with a complete vector norm that is also a ring norm, that is, $\|ab\|\leq \|a\|\|b\|$ for all $a, b\in A$. We say that $A$ is unital if it is unitary, that is, it has a unity ${\bf 1}\in A$, and $\|{\bf 1}\| = 1$. In this situation, according to the Hahn-Banach Theorem, there exists a continuous linear functional, which we will denote by ${\bf 1^*}\in A^*$, such that $\|{\bf 1^*}\| = 1$ and ${\bf 1^*}({\bf 1}) = 1$. Then, the mean functional is precisely ${\bf 1^*}$, that is, defined by

The mean operator is defined as

Finally, the centering operator is defined as

Following (3.1.2), the variance of an element $a\in A$ can be defined by

Notice that this generalization to unital Banach algebras presents a weakness: The existence of the functional ${\bf 1^*}\in A^*$ is guaranteed by the Hahn-Banach Theorem, but its uniqueness is not guaranteed. In fact, in many unital Banach algebras, such as $\ell_\infty(\Lambda)$ for instance, ${\bf 1}$ is not a smooth point of ${\mathsf{B}}_{\ell_\infty(\Lambda)}$ [16,Theorem 2.9], and thus there are infinitely many functionals of norm $1$ attaining their norm at ${\bf 1}$. It seems not trivial to overcome this issue. Maybe an option is to try to renorm equivalently the unital Banach algebra in such a way that ${\bf 1}$ becomes a smooth point of the new unit ball of the algebra or, at least, to find another smooth point in the unit ball. According to [16,Theorem 2.1], the canonical unit vector $e_{\lambda}$ is a smooth point of ${\mathsf{B}}_{\ell_\infty(\Lambda)}$ for each $\lambda\in\Lambda$. Another possibility may rely on constructing the mean functional in a $C^*$-algebra. Recall that a $C^*$-algebra is a Banach algebra $A$ endowed with an antimultiplicative and antilinear involution $*:A\to A$ satisfying that $\|aa^*\| = \|a\|^2$ for all $a\in A$.

4.3.   Generalization to probability spaces

A probability space is a $3$-tuple $(\Omega, \Sigma, \mathbb{P})$ where $(\Omega, \Sigma)$ is a measurable space and $\mathbb{P}:\Sigma\to [0, 1]$ is a probability measure, that is, a countably additive positive measure such that $\mathbb{P}(\Omega) = 1$. If $X$ is a Banach space, by $L^1((\Omega, \Sigma, \mathbb{P}), X)$ we denote the Banach space of all absolutely integrable functions, that is,

endowed with the norm

For each $f\in L^1((\Omega, \Sigma, \mathbb{P}), X)$, the mean of $f$ is defined as

The mean functional is given by

Notice that the mean functional is actually an operator, but we keep calling it functional not to mistake it with the mean operator, which will be defined next. It can be easily shown that the mean functional has norm equal to $1$. Indeed,

for every $f\in L^1((\Omega, \Sigma, \mathbb{P}), X)$, meaning that $\|\mu\|\leq 1$. Now, if we choose any $x\in {\mathsf{S}}_X$, then $x \chi_\Omega\in {\mathsf{S}}_{L^1((\Omega, \Sigma, \mathbb{P}), X)}$, and

Hence,

This proves that $\|\mu\| = 1$ and $x\chi_\Omega \in \mathrm{suppv}(\mu)$. The mean operator is then defined as

and the centering operator is

Finally, if $X$ is a unital Banach algebra, then the natural way of defining the variance of $f\in L^1((\Omega, \Sigma, \mathbb{P}), X)$ is

5.   Conclusions

It is well known in the literature of the Geometry of Banach Spaces that Hilbert spaces are transitive Banach spaces, meaning that every two elements of the unit sphere of a Hilbert space can be transported one into another by means of a surjective linear isometry. This fact confers Hilbert spaces with a certain freedom when it comes to choosing the convex series that defines the mean functional, the mean operator, the centering operator and the standard deviation. Theorem 3.1, Lemma 3.2 and Theorem 3.4 can be transported to the more general scope provided by separable Hilbert spaces discussed in the previous section. In fact, in the Discussion we unveiled how to extend the mean functional, the mean operator, the centering operator and the variance to spaces of absolutely integrable functions defined on a probability space and valued on a unital Banach algebra.

On the other hand, the study of Principal Component Analysis through Supporting Vector Analysis is a revolutionary trend that allows one to look at these Statistical concepts from a Functional Analysis viewpoint, which is more general and works in infinite dimensional environments, making possible applications in very specific settings such as, for instance, Quantum Mechanical Systems.

Acknowledgments

This work has been partially supported by the Research Grant PGC-101514-B-I00 awarded by the Ministry of Science, Innovation and Universities of Spain and co-financed by the 2014-2020 ERDF Operational Programme and by the Department of Economy, Knowledge, Business and University of the Regional Government of Andalusia under Project reference FEDER-UCA18-105867. The APCs have been paid by the Department of Mathematics of the University of Cadiz.

Conflict of interest

The authors declare that they have no conflict of interest.

Supplementary: Python GUI code

Let $A\in\mathcal{M}_{m\times n}(\mathbb{R})$ be a general matrix. The algorithm to compute the supporting vector with its corresponding first principal component, the second eigenvector with its corresponding second principal component and the next eigenvectors and principal components of $A$ is the following:

One of the novelties of this work is to apply the PCA method with a different procedure specifically, using an algorithm based on the mathematical idea that the second eigenvector, with its associated second principal component, is the supporting vector of the original points projected to the orthogonal complement of the original supporting vector. Consequently, all the principal components can be computed in an iterative process via a supporting vector.

Let $x\in\mathbb{R}^n$ be a vector. First, we show the following function to calculate the orthogonal complement of $x$:

Next, by using the previous function, we calculate the orthogonal complement of the supporting vector, representing a hyperplane with dimension $n-1$. Afterwards, the points of the original matrix, used for the initial supporting vector, are projected to this hyperplane. Hence, a new matrix is derived, whose first principal component corresponds to the second principal component of the original matrix. Thus, by applying this iterative algorithm, we can calculate all the principal components via the orthogonal complement of a supporting vector.

It is easier to understand this algorithm considering a special case. Let $A\in\mathcal{M}_{3\times 3}(\mathbb{R})$ be a matrix. In an intuitive manner, each row of the matrix represents points of $\mathbb{R}^3$. Now, we present a process of computing the first eigenvector (supporting vector) and its corresponding first principal component as before, but with a different procedure to obtain the followings. As mentioned, we notice that the first principal component via a supporting vector of a derived matrix from the original one coincides with the second principal component of the original matrix. This derived matrix is constructed projecting the original points of the original matrix to a plane formed by the orthogonal complement (two vectors) of the initial supporting vector (one vector) and the origin point. Therefore, we obtain the same result by applying the PCA presented before and this algorithm determining the next eigenvector as a supporting vector of the original supporting vector.