Explore
For Enterprise
Join for Free
Log In

艺术与人文
商务
计算机科学
数据科学
信息技术
健康
数学和逻辑
个人发展
物理科学与工程
社会科学
语言学习
學位
證書

瀏覽 Coursera 的全部課程

瀏覽
搜索
企業版
登錄
免費加入

Resilient Distributed Datasets

Loading...

Hadoop 平台和应用程序框架

加州大学圣地亚哥分校

3.9（2,854 個評分） | 120K 名學生已註冊

********* A new, improved version of the Big Data Specialization will become available on June 6! As such, enrollment for this course and all courses in this original Big Data Specialization will close on June 6. The original Big Data Specialization will continue to run until September 2016, when the Capstone will be offered for learners in this version of the Specialization. If you are in the middle of the Specialization and have purchased the entire original Big Data Specialization before June 6, Coursera will reach out to you to offer you the option of staying in the original Specialization or taking the new version. If you are just getting started on this Specialization, we recommend that you wait until June 6 to enroll in the new version. ********* This course is for novice programmers or business people who'd like to understand the core tools used to wrangle and analyze big data. With no prior experience, you'll have the opportunity to walk through hands-on examples with Hadoop and Spark frameworks, two of the most common in the industry. You will be comfortable explaining the specific components and basic processes of the Hadoop architecture, software stack, and execution environment. In the assignments you will be guided in how data scientists apply the important concepts and techniques, such as Map-Reduce that are used to solve fundamental problems in big data. You'll feel empowered to have conversations about big data and the data analysis processes.

查看授課大綱

您將學習的技能

Python Programming, Apache Hadoop, Mapreduce, Apache Spark

審閱

3.9（2,854 個評分）

5 stars
1,229 ratings
4 stars
825 ratings
3 stars
374 ratings
2 stars
203 ratings
1 star
223 ratings

YN

May 23, 2016

Easy into into big data architecture with minimal previous development requirements. Relevant to anyone who wants to grasp basic concepts of how data flows/processed/analyzed in hadoop.

DA

Mar 12, 2017

A very nice course covering the basics of the Hadoop ecosystem and Apache spark. The lectures are high quality and the presenters do a very good work of explaining the concepts. Thanks

從本節課中

Spark

Welcome to module 5, Introduction to Spark, this week we will focus on the Apache Spark cluster computing framework, an important contender of Hadoop MapReduce in the Big Data Arena. Spark provides great performance advantages over Hadoop MapReduce,especially for iterative algorithms, thanks to in-memory caching. Also, gives Data Scientists an easier way to write their analysis pipeline in Python and Scala,even providing interactive shells to play live with data.

Resilient Distributed Datasets10:41

Spark Transformations10:18

Wide Transformations10:49

教學方

Natasha Balac
Director, Predictive Analytics Center of Excellence (PACE)
Paul Rodriguez
Research Programmer
Andrea Zonca
HPC Applications Specialist

腳本

Welcome back, in this video,

we will be talking about the Resilient Distributed Datasets.

Resilient distributed datasets, usually called RDD for

short, are the data containers so are the way in spark to store your data.

So let's start from the definition and check one term at a time.

So the first is dataset, okay?

So this is like a variable or

an object in programming, so it is created,

generally reading from some disk.

So that can be HDFS or S3.

It's very convenient also if you are running from your local laptop,

you can get some local text files into Spark or

even a hierarchy of folders.

And once Spark reads those data in,

you can reference to those data with an RDD.

And the other way is by transforming another RDD.

RDDs are immutable, so you cannot change just a section of them.

But every time you transform one RDD and create a new one,

and by a series of many transformation,

you can write your data analysis part one.

The second term is distributed.

So, of course, we are running Distributed across a number of machines.

Maybe hundreds of instances on Amazon, and

the complexity is hidden by this very simple interface.

Where we can send comments if we want to execute the but

we just reference this single object.

And also data are divided in partitions.

For example, if you are working in text files, Spark doesn't work line by line,

but in group of tens or hundreds of lines, and this can be changed

depending on the application so that you can optimize performance.

And this way, partitions are divided

across all of your machines and

are the atomic chunks that are processed by your analysis biplane.

The last element is resilient and it is very important because

in the cloud environment it is pretty common to have node failures or

vary flow processes, so it's very important to be able to recover

from this situation without losing any work already done.

The technique by Spark is to track the history of each partition.

So, every point in your calculations, spark knows which

are the partitions needed to recreate the partition in case it gets lost.

And if that happens, then spark automatically figures out

where it can start from to recompute what's

the minimum amount of processing needed to recover the lost partition.

So let's now go to our console.

So you can open PySpark on the Cloudera VM.

And let's see how you can play interactively with RDD.

So a very interesting functions from start is paralyzed,

this is just used for testing purposes.

But basically it takes a data structure generally a list

on your driver program, so the driver program is your PySpark console and

distributes the across your class, there, with some number of partitions.

In this case the number three is the number of partitions,

and this sc.parallelize

function gives you back a reference to your RDD.

And then for example, one operation, the simplest operation we can do

is just collect our data back to the driving program.

And this is what you generally do once you have completed all your calculations.

So your final results are gonna be, you will want to copy them back to

your session so that you can and you can see the results.

And of course, you need to make sure that you're not trying to collect

a very big RDD, because this needs to fit in memory on your driver program.

And so in this case,

the output of this operation is the initial list because we copied it over.

So this hides the complexity

of a distributed operation where Spark has communicated with all the nodes and

asked them to communicate back their section of the RDD.

Another operation we can do is sometimes we want to check

exactly how the data are partitioned across our cluster of nodes.

And you can achieve that with the glom function.

So you can call glom and then collect.

And in this case, we get back our data but

they are actually separated by partitions.

So for the dog in park, this is a pretty interesting function.

Then we can go into reading a textFile.

For example, in the last module we saw an example where we created

a test file called testfileone in our home on cloudervian.

So you can read it either as a local file by specifying that,

thanks to the file,

a semicolon and three forward slash syntax, or

you can read it from HDFS with the syntax there that you see below.

And one simple operation you can do with this, RDD,

which is now the text_RDD, is take which copies

back to your driver program the first element of your data set.

In this case, it's going to be the first line.

So let's repeat the word count example that an Hadoop mad produce.

Using Python and using Hadoop streaming interface, now in Spark.

So the first type is the mapping phase.

The mapping phase, we were actually doing two different operations.

We were splitting each line into words and then we were creating

key value pairs where the key is the word and the value is one.

Okay.

And this was the output of mapper.pi in the Hadoop MapReduce example.

So here, we can implement those two operation as a very simple function.

So you see the first is split_words function and the second is create_pair.

And then we can apply them to our input

RDD is we flat map first, so

flat map is the same operation as a map so

applies your function to each of the elements of our RDD.

But in this case, the output of split_words is gonna be a list of words.

And we want to flatten it out so

that our output is just an RDD with all of our words.

And so this is exactly what flatMap does.

And then the output of this is gonna be a very long list of words,

and then we apply the create_pair to create the pairs word and 1.

And thanks to the fact that PySpark is very interactive.

We can take a look at this exactly what this intermediate stage does.

So we can call it collect operation on this pair RDD and this the output.

So you see for each word in our input text file.

We have a key value pair with one as the value.

Then we want to sum all of those.

Okay.

So the operation we want to do is a sum so

we implement a simple function which is sum counts.

And then what we want to do is we want to apply this but

we want to apply this key by key so we want all our instances for

example of the word far to be sum together.

In this case, the result is gonna be two.

So we call the reduceByKey operation and

we give it the reduction function, which is sum_counts,

and this is actually the output of our word count example.

Then we can collect to take a look at this on our driver program,

and this is the result, which is the same we were getting with Hadoop MapReduce.

And next time, we will be talking more in detail about all different

kinds of operations that you can apply to the data seen to flat map in map and

all the actions that he can apply to the data like reduce [INAUDIBLE].

See you next time.

探索我們的目錄

免費加入並獲得個性化推薦、更新和優惠。

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2019 Coursera Inc. 保留所有權利。

通過 App Store 下載

通過 Google Play 獲取

Coursera
關於
管理團隊
工作機會
目錄
證書
MasterTrack™ 證書
學位
企業版
政府版

社區
Learners
Partners
Developers
Beta Testers
Translators

連接
博客
Facebook
領英
Twitter
Instagram
YouTube
技術博客

更多
條款
隱私
幫助
內容訪問
媒體
聯繫我們
目錄
附屬公司