0:00

So, in the last video we introduced two parameterizations for the Q function.

One in terms of matrix WT and another in terms of

vector U_W which was given by a product of matrix WT and vector phi of basis functions.

We saw that if we somehow observed

optimal actions HE star and optimal Q function values QT star,

this would show us two equations for three unknown components of vector U_W.

Now, this suggests that a right solution,

the one that would be found by the dynamic programming method if the dynamics are known,

should be in the space of solutions parameterized by

a time-dependent matrix WT as we did in our specification of the Q function.

So, we have to find a way to learn the matrix of coefficients WT.

Let's see how this can be done.

What we need to do is to rearrange terms in

our definition of the Q function to convert it into

a product of a parameter vector and

the vector that depends on both the state and the action.

So, how we do this.

We do it in several simple steps.

First, let's write explicitly the value of the Q function as

a trace of the matrix expression given here.

We write it as the element y is product of elements of matrix W with

elements of the matrix that is formed by direct product of vectors A and phi.

This gives us the second line in this expression.

We have two types of multiplication here.

First, this symbol stands for the element y is product of two matrices,

this product is also known as the Hadamard product of two matrices.

Second, this other expression has

this circle crossed product sign and this sign stands for

a matrix given by the outer product of two vectors A_t and Phi_t.

By definition, the element ij of this matrix is given by a product of A_ti and Phi_TJ.

So, so far so good,

we converted the whole expression into a trace of a product of two matrices.

But now, we can do one more thing and represent

this expression as a scalar product of two vectors.

What we have to do to this sense is to convert

both matrices entering this expression to vectors.

We can do this by concatenating columns of

matrix W and the matrix given by the outer product of A_t and Phi_t,

and these will give us the third line in this equation.

Now, we can compactly write this whole expression as

a dot product of vector W and vector psi.

And here, vector psi is obtained by concatenating in

the columns of the outer product of vectors A_t and Phi_t.

This can be viewed as a new set of basis functions that depend both on state and action.

The dependence on the action A_t is quadratic but the dependence on the state X_t

can be arbitrary and depends on a functional form

of the original basis functions Phi_N for the state space.

If we use for example cubic D splines as

basis functions as we did in the dynamic programming solution to the model.

Then the X dependence will be a local cubic.

Now, let's compare what we got with the setting of dynamic programming.

In our current Reinforcement Learning setting,

we reduce the expression for the Q function to a dot product of

vector W and the new vector of state action basis function Psi_t.

Both have a number of components given by 3M,

so that the number of unknown coefficients in this problem is 3M as you could also

easily guess just by looking at the matrix W_t which has dimension three by M. So,

we have 3M unknowns here,

and now what about the number of observed variables?

For each time step in the batch mode reinforcement learning setting,

we have 3M observables because we observe the state X_t,

the action A_t and the reward R_t for all

of N historical or Monte Carlo pass for the stock.

So, we have a 3N observables for 3M unknowns.

And if N is larger than M,

then our problem is well-posed,

we have N divided by M observations per one phi parameter in this setting.

Now, we can compare these with

the situation we had with the dynamic programming approach.

In that case, we had to find two quantities,

the optimal policy and the optimal Q function.

In our dynamic programming solution,

we expended both in M basis functions,

so we had 2M unknown coefficients in total.

This is less than 3M unknown coefficients in the Reinforcement Learning setting.

But on the other hand,

the number of observations per times slice in

a dynamic programming approach was also less than in Reinforcement Learning approach.

And this is because in the dynamic programming setting,

we only absorb the states.

So, for N Monte Carlo pass,

we only had N data points each time.

The number of observations per parameter was therefore N divided by 2M which is

actually less than the ratio of N divided

by M that we have for the Reinforcement Learning setting.

So, it appears that at least when the data for

Reinforcement Learning is generated from the dynamic programming solution,

both the reinforcement learning and dynamic programming methods

should perform about equally well.

This setting is called on-policy learning and it means that action is

required for training of

the Reinforcement Learning occasions were actually optimal actions.

It turns out that this is indeed the case and

both algorithms produce nearly identical results for such setting.

This actually gives us a simple way to benchmark

our Reinforcement Learning algorithms because we can't so far model,

when it's fully specified by means of dynamic programming,

we can compute the optimal policy from this solution,

then we can just use the absorbed actions and

the words obtained with this policy as data for

Fitted Q-iteration method and compute

the optimal policy and Q function using this method.

Because this solution is already known from dynamic programming,

we can use it to benchmark

our Reinforcement Learning solution and check the speed of convergence,

the final results, and so on.

This is actually very ensuring as we now know exactly what we expect to see,

so any problems with Fitted Q-iteration would immediately show up if found there.

If we now want to try some other Reinforcement Learning algorithm,

instead of Fitted Q-iteration.

Again, we have a good benchmark obtained with a dynamic programming approach.

Therefore, we can test

many different Reinforcement Learning algorithms with

our simple MGP model for option pricing.

Let's take a short pause here and continue in the next video with

the actual solution of the model with the Fitted Q-iteration method.