Other interpretations of PCA (optional)

Loading...

來自 Imperial College London 的課程

Mathematics for Machine Learning: PCA

681 個評分

Imperial College London

Mathematics for Machine Learning: PCA

681 個評分

課程 3（共 3 門，Specialization Mathematics for Machine Learning）

This intermediate-level course introduces the mathematical foundations to derive Principal Component Analysis (PCA), a fundamental dimensionality reduction technique. We'll cover some basic statistics of data sets, such as mean values and variances, we'll compute distances and angles between vectors using inner products and derive orthogonal projections of data onto lower-dimensional subspaces. Using all these tools, we'll then derive PCA as a method that minimizes the average squared reconstruction error between data points and their reconstruction. At the end of this course, you'll be familiar with important mathematical concepts and you can implement PCA all by yourself. If you’re struggling, you'll find a set of jupyter notebooks that will allow you to explore properties of the techniques and walk you through what you need to do to get on track. If you are already an expert, this course may refresh some of your knowledge. The lectures, examples and exercises require: 1. Some ability of abstract thinking 2. Good background in linear algebra (e.g., matrix and vector algebra, linear independence, basis) 3. Basic background in multivariate calculus (e.g., partial derivatives, basic optimization) 4. Basic knowledge in python programming and numpy Disclaimer: This course is substantially more abstract and requires more programming than the other two courses of the specialization. However, this type of abstract thinking, algebraic manipulation and programming is necessary if you want to understand and develop machine learning algorithms.

從本節課中

Principal Component Analysis

We can think of dimensionality reduction as a way of compressing data with some loss, similar to jpg or mp3. Principal Component Analysis (PCA) is one of the most fundamental dimensionality reduction techniques that are used in machine learning. In this module, we use the results from the first three modules of this course and derive PCA from a geometric point of view. Within this course, this module is the most challenging one, and we will go through an explicit derivation of PCA plus some coding exercises that will make us a proficient user of PCA.

Steps of PCA4:07

PCA in high dimensions5:48

Other interpretations of PCA (optional)7:42

Summary of this module0:42

This was the course on PCA0:56

與講師見面

Marc P. Deisenroth
Lecturer in Statistical Machine Learning
Department of Computing

We derive PCA from the perspective of

minimising the average squared reconstruction error.

However, PCA can also be interpreted from different perspectives.

In this video, we'll have a brief look at some of these interpretations.

Let's start with a recap,

of what we have done so far.

We took a high dimensional vector x and we projected it onto

a lower dimensional representation z using the matrix B transpose.

The columns of this matrix B are the eigenvectors of

the data covariance matrix that are associated with the largest eigenvalues.

The z values are the coordinates of

our data point with respect to the basis vectors which span the principal subspace,

and that is also called the code of our data point.

Once we have that lower dimensional representation z,

we can get a higher dimensional version of it by using the matrix B again,

so multiplying B onto z to get

a higher dimensional version of the z in the original data space.

We found the PCA parameters such that the reconstruction error between x and

the reconstruction x tilde is minimised.

We can also think of PCA as a linear autoencoder.

An autoencoder encodes a data point x and

tries to decode it to something similar to the same data point.

The mapping from the data to the code is called an encoder. Let's write this down.

So this part here is called an encoder,

the mapping from the code to the original data space is called the decoder.

If the encoder and decoder are linear mappings,

then we get the PCA solution when we minimise the squared autoencoding loss.

If we replace the linear mapping of PCA with a nonlinear mapping,

we get a nonlinear autoencoder.

A prominent example of this is a deep autoencoder with

the linear functions of the encoder and decoder are replaced with deep neural networks.

Another interpretation of PCA is related to information theory.

We can think of the code as a smaller compressed version of the original data point.

When we reconstruct our original data using the code,

we don't get the exact data point back,

but a slightly distorted or noisy version of it.

This means that our compression is lossy.

Intuitively, we want to maximise

the correlation between the original data and the lower dimensional code.

More formally, this would be related to the mutual information.

We would then get the same solution to PCA we discussed

earlier in this course by maximising the mutual information,

a core concept in information theory.

When we derived PCA using projections,

we reformulated the average reconstruction error loss as minimising

the variance of the data that is projected onto

the orthogonal complement of the principle subspace.

minimising that variance is equivalent to mazimising

the variance of the data when projected onto the principle subspace.

If we think of variance in the data as information contained in the data,

this means that PCA can also be

interpreted as a method that retains as much information as possible.

We can also look at PCA from the perspective of a latent variable model.

We assume that an unknown lower dimensional code z generates data

x and we assume that we have a linear relationship between z and x.

So, generally, we can then write that x is B times z plus mu and maybe some noise.

We assume that the noise is

isotropic with mean zero and covariance matrix sigma squared times I.

We further assume that the distribution of the z is a standard normal so P of z is

Gaussian with mean zero and covariance matrix the identity matrix.

We can now write down the likelihood of this model.

So, the likelihood is P of x given z and that is a Gaussian distribution in x

with mean Bz plus

mu and covariance matrix sigma squared I.

And we can also compute the marginal likelihood as P

of x is the integral of P of x given z.

So, that is the likelihood times the distribution on z, dz,

and that turns out to be a Gaussian distribution in x with

mean mu and with covariance matrix B times B transpose plus sigma squared I.

The parameters of this model

are mu, B, and sigma squared.

And we can write them explicitly down in our model up here.

So model parameters are B and mu and sigma squared.

We can now determine the parameters of this model

using maximum likelihood estimation,

and we will find that mu is the mean of the data and B is

a matrix that contains the eigenvectors that correspond to the largest eigenvalues.

To get the low dimension code of a data point,

we can apply Bayes' theorem to invert the linear relationship between z and x.

In particular, we are going to get P of z

given x as P of x given z.

So, that is the likelihood which comes from here times P of z.

So, that's our distribution that we have here.

Divided by the marginal likelihood P of x which comes from here.

In this video, we looked at five different perspectives of

PCA that lead to different objectives: minimising the squared reconstruction error,

minimising the autoencoder loss,

mazimising the mutual information,

mazimising the variance of the projected data,

and mazimising the likelihood in a latent variable model.

All these different perspectives give us the same solution to the PCA problem.

The strengths and weaknesses of individual perspectives become

more clear and important when we consider properties of real data.

探索我們的目錄

免費加入並獲得個性化推薦、更新和優惠。

Coursera 致力於普及全世界最好的教育，它與全球一流大學和機構合作提供在線課程。

© 2019 Coursera Inc. 保留所有權利。

通過 App Store 下載

通過 Google Play 獲取