Gradient Descent Updater Strategies - Introduction to deep learning | Coursera

瀏覽 Coursera 的全部課程

瀏覽
頂級課程
登錄
免費加入

Gradient Descent Updater Strategies

Loading...

Applied AI with DeepLearning

4.4（865 個評分） | 40K 名學生已註冊

課程 3（共 4 門，Advanced Data Science with IBM 專項課程）

>>> By enrolling in this course you agree to the End User License Agreement as set out in the FAQ. Once enrolled you can access the license in the Resources area <<< This course, Applied Artificial Intelligence with DeepLearning, is part of the IBM Advanced Data Science Certificate which IBM is currently creating and gives you easy access to the invaluable insights into Deep Learning models used by experts in Natural Language Processing, Computer Vision, Time Series Analysis, and many other disciplines. We’ll learn about the fundamentals of Linear Algebra and Neural Networks. Then we introduce the most popular DeepLearning Frameworks like Keras, TensorFlow, PyTorch, DeepLearning4J and Apache SystemML. Keras and TensorFlow are making up the greatest portion of this course. We learn about Anomaly Detection, Time Series Forecasting, Image Recognition and Natural Language Processing by building up models using Keras on real-life examples from IoT (Internet of Things), Financial Marked Data, Literature or Image Databases. Finally, we learn how to scale those artificial brains using Kubernetes, Apache Spark and GPUs. IMPORTANT: THIS COURSE ALONE IS NOT SUFFICIENT TO OBTAIN THE "IBM Watson IoT Certified Data Scientist certificate". You need to take three other courses where two of them are currently built. The Specialization will be ready late spring, early summer 2018 Using these approaches, no matter what your skill levels in topics you would like to master, you can change your thinking and change your life. If you’re already an expert, this peep under the mental hood will give your ideas for turbocharging successful creation and deployment of DeepLearning models. If you’re struggling, you’ll see a structured treasure trove of practical techniques that walk you through what you need to do to get on track. If you’ve ever wanted to become better at anything, this course will help serve as your guide. Prerequisites: Some coding skills are necessary. Preferably python, but any other programming language will do fine. Also some basic understanding of math (linear algebra) is a plus, but we will cover that part in the first week as well. If you choose to take this course and earn the Coursera course certificate, you will also earn an IBM digital badge. To find out more about IBM digital badges follow the link ibm.biz/badging.

查看授課大綱

您將學習的技能

Machine Learning, Deep Learning, Long Short-Term Memory (ISTM), Apache Spark

審閱

4.4（865 個評分）

5 stars
64%
4 stars
22%
3 stars
8%
2 stars
4%
1 star
3%

KV

Apr 17, 2020

Awesome!...Exciting!.Thanks for such an interesting hands on course..really appreciate all the tutors for all the valuable knowledge and helpful responses.

BS

Aug 08, 2019

Gave a good hands-on with IBM Watson studio notebooks. Also a good overview of LSTM's, Keras, Predictive maintenance. Good stuff, keep it going

從本節課中

Introduction to deep learning

Linear algebra6:03

Deep feed forward neural networks12:57

Convolutional Neural Networks4:34

Recurrent neural networks1:46

Auto encoders and representation learning2:39

Methods for neural network training8:23

Gradient Descent Updater Strategies6:39

How to choose the correct activation function3:08

The bias-variance tradeoff in deep learning3:32

教學方

Romeo Kienzler
Chief Data Scientist, Course Lead
Niketan Pansare
Senior Software Engineer
Tom Hanlon
Training Director
Max Pumperla
Deep Learning Engineer
Ilja Rasin
Data Scientist

以免費的價格試聽課程

腳本

So welcome to the chat about gradient descent updater strategies. This is a very important lecture, because choosing the correct gradient descent updater strategy might heavily influence your learning. So remember in creating descent, we are testing one random value, because that random value is generated by the random initialization of the weight. And from there, you have to find the next step. So next step, you find of course by computing the first derivative of the cost function and you go in the direction of the steepest descent, but there are some catches. So let's start with great descent as it is. So in gradient descent, you update the weights theta by subtracting a value from it. So the value which you are subtracting is this one, so let's have a look how this is computed. Again, it starts with a cost function J. The cost function J is of course, dependent on theta, the weights, but also on the training data. So the training data, I have exemplified here using a matrix X and the matrix Y. So this is basically the complete data set. And once this cost function is computed under complete data set, we take the first derivative. We get the gradient, then we multiply by the learning rate. So the update will be smaller and then we subtract from the actual value of theta t and that gives us theta t+1. So that's how gradient descent works. So there's a variation, this is called stochastic gradient descent. And the only difference is that you are not computing the gradient under complete matrix x and the complete matrix y, you just take one training example. This is x(i) and y(i). So once you've completed the gradient, you update already the parameters theta. So a variation of this is the so-called mini batch gradient descent. So that's somehow in between. You don't use the complete data set and just don't use a single instance of your data. You just use a batch. So the batch size we have to define, usually you take values between 32 and thousand 24 or so and then you compute the gradient for that particular batch. And once this gradient has been computed, you update the parameter matrix theta. So now, another way of doing this is called momentum. So momentum, the idea is that we take also an update of a pastime step into consideration for computing the update of the current time step. So now, we have here a variable called nu. So nu computed from nu t-1 and usually we take a gamma of 0.9. And from this, basically subtract the actual gradient. Once you've computed nu t, we subtract nu t from theta t and we get the updated theta t+1. So there is a variation of momentum. This is called a Nesterov accelerated gradient. And the only addition here is that in the cost function, we subtract this term here already from theta t. So when we should be this is like a ball rolling down the hill and it's a smart ball. So whenever the slope starts to increase, the ball stops accelerating and breaks a bit. So Adagrad which stands for adaptive gradient tries to change the learning rate over time and not only for a complete patch. It tries to change the learning rate for each individual example. So you see here, this formula is dependent on i. So i stands for the actual training example. We see here that the learning rate theta is modified and it's modified by this term here. So Gt, ii is the matrix which contains information on the pass gradients per training example and taking this into account. It just reduces the learning rate. And note that there is a little term E, so that we avoid division by zero. This whole thing cannot only be computed per example, but also using matrix multiplication. And therefore, we can omit the i, because GT is a diagonal matrix. So don't worry, if you don't understand this. So maybe you have to revisit the linear algebra Basics. But anyway, it's just a way to compute the whole math in just one go. So Adadelta is only a variation of Adagrad and differences that it doesn't use a matrix G, but it continuously computes the mean of the historic gradients. So there are many, many other variations of gradient descent updater and I recommend you to have a look at Sebastian's blog later where everything is described really nicely. But the key take home point here is that the gradient descent updater strategy is a very important job, you have to tune. And as usual, tuning neural network is considered as a split magic or trial and error. So you just have to try a couple of those. So let's actually have a look at Sebastian's blog. So the best thing is this funny guy here. And you see here, those are the trajectories of different learning curves using different gradient descent updater strategies. So it's pretty interesting, because it's a very important type of parameter you can tune. So we have covered most of those, but what I wanted to show you is two little figures here. So those two are really interesting. So you see here different trajectories in optimization space using different gradient descent updater. So the red one here is stochastic gradient descent. And to see here in both problems, it performs really poor and even get stuck. So maybe in the first image, it won't get stuck, but it won't converge for ages. And in the second, this is a saddle point, it even gets stuck. And you can see here that, for example, a delta is performing best among all of those. Okay, I think that's enough for now. I hope that you know understood that this is an important type of parameter which you can tune and that's basically it.

探索我們的目錄

免費加入並獲得個性化推薦、更新和優惠。

熱門在線課程

適用於所有人的人工智能課程
TensorFlow 簡介
神經網絡與深度學習
算法第 1 部分
算法第 2 部分
機器學習
使用 Python 進行機器學習
使用 SAS Viya 進行機器學習
R 語言程序設計（中文版）
Matlab 編程入門
使用 Python 進行數據分析
AWS 基礎知識：走向雲原生
Google 雲端平台基礎知識
網站可靠性工程
英語專業說
幸福科學
選修“如何學習”這門課
金融市場
公共健康領域的假設檢驗
日常領導基礎知識

熱門在線專項課程

深度學習
零基礎 Python 入門
數據科學
借助 Python 應用數據科學
商務基礎
Architecting with Google Cloud Platform
Google 雲端平台數據工程
Excel 到 MySQL
高級機器學習
數學在機器學習領域的應用
自動駕駛汽車
企業區塊鏈革命
商業分析
Excel 辦公技能
數字化營銷
使用 R 進行公共健康領域的統計分析
免疫學基礎知識
解剖學
管理創新和設計思維
積極心理學基礎知識

在線證書

Google IT 支持
IBM 客戶關係專員
IBM 數據科學
應用項目管理
IBM Applied AI 專業證書
用於分析的機器學習
空間數據分析和可視化
建築工程與管理
教學設計

在線學位課程

數據科學碩士
計算機科學學士學位
計算機科學與工程學位
機器學習碩士
MBA 和商學學位
電氣工程碩士
公共健康碩士
信息技術碩士

Coursera

關於
管理團隊
工作機會
目錄
證書
MasterTrack™ 證書
學位
企業版
政府版
面向校園
新冠疫情響應

社區

學生
合作夥伴
開發者
Beta 測試人員
專業譯員
博客
技術博客

更多

條款
隱私
幫助
內容訪問
媒體
聯繫我們
目錄
附屬公司

通過 Google Play 獲取

© 2020 Coursera Inc.保留所有權利。