Deep learning in optical flow estimation

Loading...

來自 National Research University Higher School of Economics 的課程

Deep Learning in Computer Vision

42 個評分

National Research University Higher School of Economics

Deep Learning in Computer Vision

42 個評分

課程 5（共 7 門，Specialization Advanced Machine Learning）

Deep learning added a huge boost to the already rapidly developing field of computer vision. With deep learning, a lot of new applications of computer vision techniques have been introduced and are now becoming parts of our everyday lives. These include face recognition and indexing, photo stylization or machine vision in self-driving cars. The goal of this course is to introduce students to computer vision, starting from basics and then turning to more modern deep learning models. We will cover both image and video recognition, including image classification and annotation, object recognition and image search, various object detection techniques, motion estimation, object tracking in video, human action recognition, and finally image stylization, editing and new image generation. In course project, students will learn how to build face recognition and manipulation system to understand the internal mechanics of this technology, probably the most renown and oftenly demonstrated in movies and TV-shows example of computer vision and AI.

從本節課中

Object tracking and action recognition

The fourth module of our course focuses on video analysis and includes material on optical flow estimation, visual object tracking, and action recognition. Motion is a central topic in video analysis, opening many possibilities for end-to-end learning of action patterns and object signatures. You will learn to design computer vision architectures for video analysis including visual trackers and action recognition models.

Introduction to video analysis3:39

Optical flow5:14

Deep learning in optical flow estimation5:44

Visual object tracking5:26

Examples of visual object tracking methods13:11

Multiple object tracking5:43

Examples of multiple object tracking methods8:05

與講師見面

Anton Konushin
Senior Lecturer
HSE Faculty of Computer Science
Alexey Artemov
Senior Lecturer
HSE Faculty of Computer Science

In this video I will talk about the application of deep learning into

optical flow estimation.

The straightforward way is to create a neural network that takes two

frames as input and produce optical flow map as output, simple at that.

This approach was successfully proposed in FlowNet paper in 2015.

Two versions of FlowNet architecture was proposed.

In first version,

two color frames are concatenated into one image with six channels, instead of three.

This image is then passed to the fully convolutional network.

In second version, a Siamese network is used.

Two frames are separately passed through convolutional layers for

future computation.

That features are passed to a specific layer which implements patch comparison.

It works similar to convolution, but without learnable parameters.

This layer, dot-product between feature vectors for pixel in the first image and

for pixel in the second image is computed.

The results are passed next to the series of convolutional layers

to estimate optical flow.

Obviously, current optical flow datasets provide

insufficient data to train such complex model from scratch.

So authors of FlowNet solved this problem by creating

a new synthetic dataset called flying chairs.

This dataset is rather simple compared to all previous ones.

In this data set, various chairs are flying on top of the background.

The scenes looks very unrealistic, but it provides a way to generate a lot of data.

And it proves to be enough for training the FlowNet to demonstrate accuracy

similar to the top optical flow estimation methods.

As expected,

FlowNet outperforms other existing methods on testing part of flying chairs dataset.

But the results are lower on the other dataset.

However, the validity of this simple approach is proven in this work.

Probably, if you would have enough good training data someday,

then it will obtain both fast and accurate flow estimation method with this approach.

If you can't get a good solution with one neural network model for

the whole problem, then we can integrate convolutional neural networks model

into more elaborate frameworks.

As I have mentioned previously, optical flow estimation

is essentially a pixel correspondence estimation problem, or a matching problem.

And CNN models has demonstrated to be a powerful method for pixel matching.

In computing the stereo matching cost with a convolutional neural network,

a CNN model has been proposed for dense stereo matching.

The nine by nine pixel windows are into a Siamese network for comparison.

Due to the small window size,

we can collect a lot of training examples from each pair of images.

We can create a positive pair of matching windows from pixel in the first image and

from pixel in the second image, which are close to the true correspondence.

All pixels in the second image which are farther from correspondence

can give us a negative matching pair.

The dense matching of original input images can be breaked down into set of

simpler problems with multi-scale matching.

First, we downscale input images.

Then we apply dense matching to the downscaled version of input images,

which is a simpler problem.

For downscaled image, pixel displacement between images are shorter,

thus the search space is smaller.

The number of pixels in downscaled images are also much smaller.

So we can apply complicated matching procedures that requires global

optimization methods.

Then optical flow from downscaled images can be used as a starting point for

optical flow estimation on original input images, which is much simpler than to try

to estimate optical flow from original images.

Many recent state of the art optical flow estimation methods use this approach.

For example, in EpicFlow,

edge-preserving guided interpolation is applied to sparse matches between images.

As of July 2017, the best optical flow estimation methods combined pixel

matching by CNN model is guided upscale.

In this method, first pixels are mapped to feature vectors this CNN model.

This model is similar to the model for stereonation,

which I have described previously.

The authors have demonstrated that for reasonably high quality,

it is enough to make pixel to only ten dimensional vector.

Due to the small size of each vector, you can directly compute matching cost volume

by dot-product between normalized feature vector in first image and

all possible displacement in the second image.

From this cost volume, the matches are extracted with semi-global minimization,

then guided upscaling is applied.

It can be seen from example in this slide.

The semi-dense matches obtained from cost volume

are already close to the ground truth data.

Guided upscaling of optical flow map can be performed by EpicFlow.

But, in this paper,

authors have additionally used homography based interpolation.

For this, image is first segmented and

homography transformation is fit to semi-dense matches in each segment.

If homography fits pixels of the segment well,

then this segment has simple shape and upscaling is performed this homography.

Otherwise EpicFlow is used for this segment.

In the slide, a comparison between EpicFlow interpolation and

new interpolation is demonstrated.

Experiments show that new interpolation improved the optical flow.

See parts of error maps near the lower border.

As a result, the described approach outperforms existing state of

the art methods in both speed and accuracy.

A number of other of further state of the art methods also rely on pixel matching

these CNN models on downscaled images and guided upscaling.

[MUSIC]

探索我們的目錄

免費加入並獲得個性化推薦、更新和優惠。

Coursera 致力於普及全世界最好的教育，它與全球一流大學和機構合作提供在線課程。

© 2018 Coursera Inc. 保留所有權利。

通過 App Store 下載

通過 Google Play 獲取