In the last course of our specialization, Overview of Advanced Methods of Reinforcement Learning in Finance, we will take a deeper look into topics discussed in our third course, Reinforcement Learning in Finance. In particular, we will talk about links between Reinforcement Learning, option pricing and physics, implications of Inverse Reinforcement Learning for modeling market impact and price dynamics, and perception-action cycles in Reinforcement Learning. Finally, we will overview trending and potential applications of Reinforcement Learning for high frequency trading, cryptocurrencies, peer-to-peer lending, and more.
The Fokker-Planck Equation and Quantum Mechanics
Loading...
來自 New York University Tandon School of Engineering 的課程
Overview of Advanced Methods of Reinforcement Learning in Finance
2 個評分
New York University Tandon School of Engineering
2 個評分
課程 4（共 4 門，Specialization Machine Learning and Reinforcement Learning in Finance）
從本節課中
Perception - Beyond Reinforcement Learning
與講師見面
Igor Halperin
Now, I want to continue with the Fokker-Planck Equation in log
space that we derived in the last video, and which I'll show you here once again,
along with the typical form of the potential that can arise for such problem.
Such potentials with the metal stable state and
decaying potential behind the barrier are in fact, well studied in physics,
where they describe tons of differences in distance phenomena.
In particular they describe the famous problem of
a decay of a metastable state in statistical physics,
which was solved by Kramer in 1940s.
It is for this reason that this problem is sometimes called a Kramer Escape problem.
We will talk about it in a sec but first let's discuss something more
conventional with the Fokker-Planck equation,
namely the existence of a steady state and time independent solution.
This problem is easy to solve, at least formerly.
So for a steady state the left-hand side of the Fokker-Plank equation
should vanish.
But the right-hand side of this equation is a total derivative
of a quantity called the probability current.
So if the left-hand side vanishes, we can express the Fokker-Plank
equation as a conservation of the probability current.
And the in first stationary state, also called the ground
state if the state exists then the current should vanish.
And this simple argument leads to a suggestive status
state solution in the form shown in equation 51 here.
This is what is known as Boltzmann or exponential distribution with energy U.
Exponential distribution survey popular tools in both physics and
machine learning.
And we mentioned exponential models a few times in our courses.
Here such a distribution is derived from a physics motivated model.
Now, this distribution has a normalization factor Z in front of it.
And this part is a bit tricky but also the most interesting one.
The point is that a steady state density only exists if it's normalizeable
which requires that the normalization constant Z should be finite and positive.
But if there is escape from a metastable state through tunnel link,
the system cannot have a genuine steady state.
What can be obtained instead in such a system would be a Quasi-stationary state
that might be very long lived in fact, but still will decay in a very long time.
And the time it takes for
such state to decay is driven by the height of the pillar and the noise level.
The existence of such process shows up in divergence or
zeros of the normalization factor Z.
That is called a partition function in physics.
And it turns out that such a scape and tunneling problems can also be
described using the language of quantum mechanics.
This can be done starting with the Fokker-Planck equation as follows.
First, we change the dependent variable and
try the unknown density pure white as shown in equation 52 as
a product of the exponential factor and a new unknown function K(y).
And please note that the exponential factor here is a square root of
the steady-state distribution.
Now if we plug such answers into the Fokker-Planck Equation, we get another
equation for the newer non-function K(y), and this is shown in equation 53.
This equation is actually the same as the Schrodinger equation of quantum mechanics,
where sigma squared plays the role of the Planck constant.
In quantum mechanics, the Schrodinger equation is the main equation of dynamics,
and it replaces the Newtonian, Lagrangian, and Hamiltonian methods for
the classical mechanics to the setting of quantum domains.
But in the original Schrodinger equation, there is an additional imaginary unit,
i, in the left-hand side in front of the time derivative.
And here there is none.
So this equation can be interpreted as a Schrodinger equation in imaginary time.
So as a result, we find that the classical stochastic system is
mathematically equivalent to quantum mechanics in imaginary time,
and a variance sigma square playing the role of the Planck constant.
The Hamiltonian for the result in Schrodinger equation for
this case is shown in equation 54.
And it turns out that the Schrodinger equation has many,
many interesting properties.
And one of them is called supersymmetry or SUSY as physicists call it.
Not only Schrodinger equation has supersymmetry, but
this one obtained from the classical stochastic dynamics does have it.
An the supersymmetry of the Schrodinger equation is based on the observation that
are Hamiltonian H can be factorized as a product of two operators,
A and A+, as shown in equations 55, and 56 here.
These operators are sometimes called the supercharge generators.
And the function, U prime is called the superpotential.
Now if we have these generators we can swap the order and
get a new Hamiltonian H+ shown in equation 57.
And the most interesting thing about this pair of Hamiltonians H and
H+ is that they have the generate spectra for all states excluding
the lowest energy state with zero energy, if such state exists.
And these can be seen using the simple chain of
transformations shown in equation 58.
If Si n is an eigenstate of H, who is an eigenvalue En,
then we can form a new state, A times this state, and this state
will be an eigenstate of the SUSY partner Hamiltonian H+ with the same energy.
And this means that all eigenstates of H with non zero energy
should be the degenerate with eigenstates of H+.
And now SUSY can be unbroken or spontaneously broken.
If it's unbroken, a ground state with zero energy exists.
On the other hand, if the energy of the ground state is larger than zero,
supersymmetry is broken.
And it turns out that mechanisms of breaking supersymmetry in
quantum mechanics and quantum field theory are the same mechanisms
that lead to tunneling and escape from metastable potential.
So how the escape looks like.
If we go back to the language of classical statistical physics,
then the process is described as an event when due to thermal fluctuations,
a particle gets enough energy to jump over the barrier.
And the probability of such event will be obtained as a product
of two factors, the Arrhenius factor B and pre-factor A.
The Arrhenius factor B is shown here in equation 60.
And its exponential in parameter Eb that gives the height of the barrier.
So if a barrier is very high then the actual escape probability can be
very tiny.
And vice versa, if a barrier is not too high, or
the energy of a particle is such that it's near the top of the barrier
then the escape probability might become quite noticeable.
And the remaining pre-factor A can also be computed for
one dimensional diffusion it turns out that this factor is proportional to
the frequency of oscillations, omega near the bottom of the potential well.
That's shown in equation 61 here.
It turns out that the same expression can also
be obtained from an equivalent quantum mechanical formulation.
And in this case it turns out that tunneling can also be described by
the laws of classical mechanics.
But applied in imaginary time.
In imaginary time, the kinetic energy becomes negative, and
the action becomes imaginary.
As you can see, if you look again at our equation,
for, which I did here as equation 62, for your convenience.
So for this case, the expression in the square root in there
integral is negative and therefore the action itself is imaginary.
But because the weight of the action is I times S,
this produces exponentially suppressed tunneling in quantum mechanics.
And finally, a few more words about the tunneling effect.
This effect is non perturbative as we said, so
it cannot be obtained as an expansion in small values of parameters kappa and
g around a model with a trivial vacuum x=0.
It turns out if we still start developing such
perturbative schemes they become divergence series and
the origin of these divergence of perturbative series and
tunneling turns out to be the same.
A mechanism for this is similar to divergence of Quantum Electro-Dynamics
that was discovered by Freeman Dyson in 1950s.
You can read more about such problems since statistical physics and
quantum mechanics in your weekly reading quiz for this week.
And for now we just want to summarize.
So we saw that reinforcement learning and inverse reinforcement learning can be used
not only to compute specific numbers in finals, but also to construct new models.
And we presented one such simple model for market dynamics that is inspired or
kind of derived from reinforcement learning in our previous course.
Now, in this week we took another look at the same model and
found the need to extend it by introducing the cubic non-linearity.
And this falls from the analysis of behavior of the model and
is needed for stability.
This cubic non-linearity can also probably be derived directly from
the enforcement learning approach.
But here we take a simpler route and just added this to our phenomenological grounds
using arguments based on asymptotic analysis on electricity and symmetries.
And all these are common are useful tools in physics.
In particular symmetries play a major role in determining characters or
phase transitions between different phases of matter.
Here we use such sort of analysis in a similar way to using
prior Bayesian statistics.
This is something that is not directly in the data but
should hold anyway based on some more general arguments.
In the course project for this course you will analyze the model that you
did in your previous course, but this time with keeping a non-zero value of g.
And hence with keeping a cubic non linearity.
And this will be it for this lesson.
And see you the next week.
