Vector Space

Axioms

Given a set $B$ and a function written $|| a ||$ called the norm, for every element $a, b \in B$, we say the set and the norm comprise a vector space if the following three properties hold

(N1) Positive Definite

The value of the norm is always positive, unless the argument is zero, in which case the value is zero.

$$|| a || \ge 0$$

(N2) Scalable

Given a scalar $\lambda$, multiplying the norm with the scalar is the same as taking the norm after multiplying with the scalar

$$\lambda || a || = || \lambda a ||$$

The scalar does not need to come from $B$ but it does need to come from the underlying field of $B$. For example, construct $B$from all pairs of real numbers. We can use any real number as a scalar to satisfy this requirement.

This is also known as the homogeneity property. But that term, I feel, is overloaded in various contexts. I'm sticking with the informal term "scalable" here, as it says exactly what we're doing and with what tools, eg., the scalar number.

(N3) Triangle Inequality

The triangle inequality has structural similarity to the property with the same name in metric spaces. In the context of vector spaces, however, I find it has a crisper memnonic: the norm of the sum is less than or equal to the sum of the norms.

$$|| a + b || \le || a || + || b ||$$

Pick $b = 0$. Then $|| a || \le || a ||$is true by virtue of positive definiteness. Pick $b = a$. Then $|| 2a || \le 2|| a||$is true by scalability. By this reasoning, for any choice of $a, b \in B$, the norm of the sum cannot be less than the sum of the norms.

Various Norm Functions

As with metric spaces, a vector space is described by both the set of vectors and the norm function. There are many candidate norm functions, each with distinct behaviors. Here's the most common norms.

L∞ Minimax Norm

$L_{\infty}$ picks the largest absolute value of a vector.

$$||f||_{\infty} = \max{ | f(x) | }$$

L1 Uniform Norm

$L_1$ gives the total area covered by a vector.

$$|| f ||_1 = \int^b_a | f(x) | dx$$

L2 Least Squares

$L_2$ gives the "size" of a vector. Using $R^2$ as the vector space makes $L_2$ the same as the Euclidean distance metric function.

$$||f||_2 = \biggl ( \int^b_a f(x)^2 dx\biggr)^{1/2}$$

L-p Norm

But notice each of the norms above have a very distinctive "shape" to them - there's a pattern here. That's because they're all instances of the L-p norm, which can be written

$$|| f ||_p = \bigg[ \int^b_a{|f(x)|^pdx} \biggl]^{1/p}, \space 1 \le p \lt \infty$$

Check that, substituting $p = 1, \space 2, \text{or } \infty$ gives the norm functions mentioned earlier.

The boundaries for the integral are important here as they indicate the norm is defined for continuous, real valued functions on an interval, written $C[a,b]$.

All three norm axioms (N1), (N2), and (N3) are easily demonstrated for both the minimax and uniform norms.

For least squares, use the Minkowski inequality to demonstrate (N3).

The more general norm requires the Holder inequality

Important Properties

Cauchy-Schwarz Inequality

Using the 2-norm, we state the product of norms as an inequality with the integration of vector products

$$\biggl | \int^a_b f(x)g(x)dx \biggr | \le || f ||_2 \space || g ||_2$$

Minkowski Inequality

The Minikowski inequality is just the triangle inequality for the 2-norm.

$$|| f + g ||_2 \le ||f||_2 + ||g||_2$$

We arrive at this via the Cauchy-Schwarz inequality. For more general versions of Minkowski, we need to use the Holder inequality below.

Holder Inequality

For some pair of integers where $\frac{1}{p} + \frac{1}{q} = 1$, the product of norms is greater than or equal to the 1-norm of the product, eg.,

$$\biggl | \int f(x) g(x) dx \biggr | \le || f ||_p \space || g ||_q$$

Notice that when $p = q = 2$ we have the Cauchy-Schwartz inequality.

(Strict) Convexivity

Theorems

See Normed Vector Spaces