Inner products of functions and random variables (optional)

Loading...

來自 Imperial College London 的課程

Mathematics for Machine Learning: PCA

681 個評分

Imperial College London

Mathematics for Machine Learning: PCA

681 個評分

課程 3（共 3 門，Specialization Mathematics for Machine Learning）

This intermediate-level course introduces the mathematical foundations to derive Principal Component Analysis (PCA), a fundamental dimensionality reduction technique. We'll cover some basic statistics of data sets, such as mean values and variances, we'll compute distances and angles between vectors using inner products and derive orthogonal projections of data onto lower-dimensional subspaces. Using all these tools, we'll then derive PCA as a method that minimizes the average squared reconstruction error between data points and their reconstruction. At the end of this course, you'll be familiar with important mathematical concepts and you can implement PCA all by yourself. If you’re struggling, you'll find a set of jupyter notebooks that will allow you to explore properties of the techniques and walk you through what you need to do to get on track. If you are already an expert, this course may refresh some of your knowledge. The lectures, examples and exercises require: 1. Some ability of abstract thinking 2. Good background in linear algebra (e.g., matrix and vector algebra, linear independence, basis) 3. Basic background in multivariate calculus (e.g., partial derivatives, basic optimization) 4. Basic knowledge in python programming and numpy Disclaimer: This course is substantially more abstract and requires more programming than the other two courses of the specialization. However, this type of abstract thinking, algebraic manipulation and programming is necessary if you want to understand and develop machine learning algorithms.

從本節課中

Inner Products

Data can be interpreted as vectors. Vectors allow us to talk about geometric concepts, such as lengths, distances and angles to characterise similarity between vectors. This will become important later in the course when we discuss PCA. In this module, we will introduce and practice the concept of an inner product. Inner products allow us to talk about geometric concepts in vector spaces. More specifically, we will start with the dot product (which we may still know from school) as a special case of an inner product, and then move toward a more general concept of an inner product, which play an integral part in some areas of machine learning, such as kernel machines (this includes support vector machines and Gaussian processes). We have a lot of exercises in this module to practice and understand the concept of inner products.

Inner product: definition5:02

Inner product: length of vectors7:07

Inner product: distances between vectors3:42

Inner product: angles and orthogonality5:41

Inner products of functions and random variables (optional)7:22

Heading for the next module!0:35

與講師見面

Marc P. Deisenroth
Lecturer in Statistical Machine Learning
Department of Computing

In the previous videos, we looked at properties of

Inner Products to compute Lengths, Angles and Distances.

We focused on inner products of finite dimensional vector spaces.

In this video, we will look at two examples of inner products of other types of vectors,

inner products of functions and inner products of random variables.

The inner products we discussed so far were

defined for vectors with a finite number of entries,

and we can think of these vectors as

discrete functions with a finite number of function values.

The concept of an inner product can be

generalized to continuous valued functions as well.

And then the sum over individual components of vectors turns into an integral,

and the inner product between two functions is defined as follows.

You can write that the inner product between two functions

u and v is the integral of an interval from a to b.

A u of x times v of x,

d x, and as with our normal inner product,

we can define Norms and Orthogonality by looking at this inner product.

If that integral evaluates to zero,

the functions u and v are orthogonal.

Let's have a look at an example.

If we choose u of x equals

Sine x and v of x is Cosine of x,

and we define f of x to v,

u of x times v of x,

which is a Sine x times Cosine x then we're going to end up with this function.

This function is Sine x times Cosine x.

We see that this function is odd,

which means that f of minus x equals minus f of x.

If we choose the integral limit to be minus Pi and plus Pi then the

integral of this product Sine of x times Cosine of x evaluates to zero.

And that means that Sine and Cosine are Orthogonal,

and actually holds that,

if you look at a set of functions say one Cosine x,

Cosine of two x,

Cosine three x and so on,

that all of these functions are Orthogonal to

each other if we integrate from minus Pi to plus Pi.

Another example for defining an inner product between

unusual types are random variables or random vectors.

If we have two random variables,

which are uncorrelated, then we know the following relationship.

We know that, the variance of x plus y is the variance

of x plus the variance of y where x and y are random variables.

Remember that variances are measured in squared units.

This looks very much like the Pythagorean Theorem for right triangles,

that one states that c squared equals a squared plus b squared.

If we look at triangles of this form where this is a,

b, and this is c. Let's see whether we can find

a geometric interpretation of the variance relation of uncorrelated random variables.

Random variables can be considered vectors in a vector space and we can

define inner products to obtain geometric properties of these random variables.

If we define the inner product between two random variables between

x and y to be the covariance between x and y,

we see that the covariance is Symmetric,

Positive Definite, and Linear.

So linearity would mean that

the covariance of lambda times x plus y and z where x,

y and z are random variables,

and lambda is a real number,

is lambda times the covariance between x and

z plus the covariance between y and z.

And if the length of a random variable is

the square root of the covariance between of x with itself,

which is the square root of the variance of x,

then this is the standard deviation of the random variable x.

So this is the length of a random variable.

Therefore, the zero vector is a vector that has no uncertainty,

that means standard deviation is zero.

If we now look at the angle between two random variables,

we get the following relationship.

We get the Cosine of theta,

which is the angle between two random variables is by

definition the inner product between the two random variables,

divided by the length of the first random variable,

times the length of the second random variable.

And if we now write this out using the definition of our inner product,

we get the covariance between x and y,

divided by the square root

of the variance of x,

times the variance of y.

And this evaluates to zero if the covariance between x and y is zero,

and that is the case when x and y are uncorrelated.

Coming back now to our geometric interpretation,

we would now replace a with the standard deviation of x,

b is the standard deviation of y,

and c is the square root of

the variance of x plus the variance of y.

And this is how we get our geometric interpretation of random variables.

In this video, we looked at inner products of rather unusual objects,

functions and random variables.

However, even with functions and random variables,

the inner product allows us to think about lengths and angles between these objects.

In the case of random variables,

we saw that the variance of the sum of two uncorrelated random variables can

be geometrically interpreted using the Pythagorean Theorem.

探索我們的目錄

免費加入並獲得個性化推薦、更新和優惠。

Coursera 致力於普及全世界最好的教育，它與全球一流大學和機構合作提供在線課程。

© 2019 Coursera Inc. 保留所有權利。

通過 App Store 下載

通過 Google Play 獲取