Optical flow

Loading...

來自 National Research University Higher School of Economics 的課程

Deep Learning in Computer Vision

59 個評分

National Research University Higher School of Economics

Deep Learning in Computer Vision

59 個評分

課程 5（共 7 門，Specialization Advanced Machine Learning）

Deep learning added a huge boost to the already rapidly developing field of computer vision. With deep learning, a lot of new applications of computer vision techniques have been introduced and are now becoming parts of our everyday lives. These include face recognition and indexing, photo stylization or machine vision in self-driving cars. The goal of this course is to introduce students to computer vision, starting from basics and then turning to more modern deep learning models. We will cover both image and video recognition, including image classification and annotation, object recognition and image search, various object detection techniques, motion estimation, object tracking in video, human action recognition, and finally image stylization, editing and new image generation. In course project, students will learn how to build face recognition and manipulation system to understand the internal mechanics of this technology, probably the most renown and oftenly demonstrated in movies and TV-shows example of computer vision and AI.

從本節課中

Object tracking and action recognition

The fourth module of our course focuses on video analysis and includes material on optical flow estimation, visual object tracking, and action recognition. Motion is a central topic in video analysis, opening many possibilities for end-to-end learning of action patterns and object signatures. You will learn to design computer vision architectures for video analysis including visual trackers and action recognition models.

Introduction to video analysis3:39

Optical flow5:14

Deep learning in optical flow estimation5:44

Visual object tracking5:26

Examples of visual object tracking methods13:11

Multiple object tracking5:43

Examples of multiple object tracking methods8:05

與講師見面

Anton Konushin
Senior Lecturer
HSE Faculty of Computer Science
Alexey Artemov
Senior Lecturer
HSE Faculty of Computer Science

[SOUND] In this video, I will talk about motion in video.

Motion is the main difference between static images and videos.

Motion itself is a powerful visual que.

And many actions are defined by motion.

Sometimes it's enough to see the motion of a set of sparse points for

recognition of object properties and actions.

For example, from dynamics point lights on parts of human body,

you can recognize gender, complexion, and mood of a person.

I recommend you to check this cool demo from Biomotion Lab to see for yourself.

Suppose points of observed scene are moving relative to the camera.

Vector field of 2D projections of scene point

motion vectors into image is called motion field.

We need to measure the motion field to obtain motion features for

subsequent recognition.

Sometimes object does move, but we can't see its motion.

Try to imagine smooth gray sphere with uniform texture

which rotates around its axis.

All points of the sphere are moving, but

we can't distinguish one point from the other.

So for us, the sphere seems static.

Optical flow is a vector field of apparent motion of pixels between frames.

Optical flow is what we can estimate from video.

We can treat optical flow as estimation of the true motion field.

Optical flow estimation is one of the key problems in video analysis.

Optical flow estimation can be regarded as a dense correspondence problem.

Let vector field u, v be the optical flow field.

To estimate optical flow field for each point x, y in first frame,

we need to find a corresponding point x + u, y + v in the second frame,

which corresponds to the same point of zone c as the point on the first frame.

There are two main ways to visualize the result of optical flow estimation.

First is to directly draw motion vectors, but in this case,

we can draw motion vectors only for a sparse set of points because, otherwise,

the image will be unreadable.

The other way is color coding.

For each possible motion vector, we specify a color.

Usually, vector orientation is coded by color hue and

vector length is coded by color saturation.

In this case, we can visualize motion vectors for each pixel in the image.

To evaluate the procedure of optical flow estimation,

we need to compare estimated optical flow field and ground truth optical flow field.

There are several metrics.

Two of the most often used are angular error and endpoint error.

Angular error is the angle between estimated optical flow vector and

ground truth optical flow vector.

It is measured as arccos of dot product of these vectors.

Endpoint error is the distance between endpoints of estimated optical flow vector

and ground truth optical flow vector.

But how to obtain ground truth optical flow?

It's very tricky problem.

Compared to image classification or

object detection, we can't just annotate image with human iterator.

And we need to specify correct optical flow vectors for every pixel in the image.

As a result, the number of available ground truth data for

optical flow estimation is very limited.

In Middleburry optical flow dataset, which was published in 2011,

three ways for generation of ground truth data were used.

First, is frame-by-frame capturing of scene in both common and

fluorescent lighting.

A lot of dot seen only in fluorescent lighting are placed on the surface of

the object from which high quality optical flow can be estimated by existing method.

Second is a fully synthetic image generation from 3D model scene.

And the third is just interpolation of video frames from high speed camera

if first and last frame in the sequence are null.

In this case, we don't know the true optical flow.

But if we can compare the interpolated video frame,

this is the true intermediate frame.

The second dataset is KITTI Vision Benchmark Suite.

It is a collection of data obtained by laser scanning of

the urban environment from a car.

Optical flow dataset is one dataset from this collection.

There are other datasets.

To obtain this dataset, the complicated procedure has been used.

3D models are separately fit to the ground scene and to each of the moving objects.

From these models, a correct optical flow between images is reconstructed.

Due to the complexity of the procedure,

the dataset contains only less than 200 image pairs.

The third contemporary optical flow ground truth dataset is based on an open source

3D movie Sintel.

For this movie, all 3D data is available.

From various parts of the movie, more than 1,000 training and

500 test frames were extracted.

Because various rendering options can be used for the same moment in the movie,

we can obtain images with different level of rendering.

For example, in the slide is demonstrated that for

the first image, no shading was used.

For the second image, complex shading and lighting are added.

For the third image, such effects as motion blur or depths of field are added.

This allow us to evaluate optical flow estimation methods in various cases.

[SOUND]

[MUSIC]

探索我們的目錄

免費加入並獲得個性化推薦、更新和優惠。

Coursera 致力於普及全世界最好的教育，它與全球一流大學和機構合作提供在線課程。

© 2018 Coursera Inc. 保留所有權利。

通過 App Store 下載

通過 Google Play 獲取